A new patient-reported outcome measure for the evaluation of ankle instability: description of the development process and validation protocol

Abstract

Background

Acute ankle sprains represent one of the most common traumatic injuries to the musculoskeletal system. Many individuals with these injuries experience unresolved symptoms such as instability and recurrent sprains, leading to chronic ankle instability (CAI), which affects their ability to maintain an active lifestyle. While rehabilitation programs focusing on sensorimotor, neuromuscular, strength and balance training are primary treatments, some patients require surgery when rehabilitation fails. A critical analysis of the patient-reported outcome tools (PROs) used to assess CAI surgical outcomes raises some concerns about their measurement properties in CAI patients, which may ultimately affect the quality of evidence supporting current surgical practice. The aim of this research is to develop and validate a new PRO for the assessment of ankle instability and CAI treatment outcomes, following recent methodological guidelines, with the implicit aim of contributing to the generation of scientifically meaningful evidence for clinical practice in patients with ankle instability.

Methods

Following the COnsensus-based Standards for the selection of Health Measurement Instruments (COSMIN), an Ankle Instability Treatment Index (AITI) will be developed and validated. The process begins with qualitative research based on face‒to‒face interviews with CAI individuals to explore the subjective experience of living with ankle instability. The data from the interviews will be coded following an inductive approach and used to develop the AITI content. The preliminary version of the scale will be refined through an additional round of face‒to‒face interviews with a new set of CAI subjects to define the AITI content coverage, relevance and clarity. Once content validity has been examined, the AITI will be subjected to quantitative analysis of different measurement properties: construct validity, reliability and responsiveness.

Discussion

The development of AITI aims to address the limitations of existing instruments for evaluating surgical outcomes in patients with CAI. By incorporating patient input and adhering to contemporary standards for validity and reliability, this tool seeks to provide a reliable and meaningful assessment of treatment effects.

Trial registration

Not applicable.

Background

Acute ankle ligament injuries are among the most common musculoskeletal injuries in both the general and sports populations [14]. A significant number of people who have suffered a first ankle ligament injury have unresolved posttraumatic symptoms lasting more than 1 year, such as feelings of instability (33–55%), recurrent episodes of ‘giving way’ and sprains (3–35%), and, in some cases, pain [17, 39]. This condition is referred to as chronic ankle instability (CAI), a multifaceted syndrome that is associated with functional and/or structural deficiencies and impaired quality of life and decreased physical activity [2, 15].

CAI patients are primarily treated with a comprehensive rehabilitation program that emphasizes ankle sensorimotor, strength and balance training. Rehabilitation has been reported to improve subjective symptoms and functional limitations and reduce the risk of ankle reinjury in CAI patients [1, 9]. However, despite prolonged functional rehabilitation, some patients with CAI continue to experience significant activity restrictions due to ankle problems. When rehabilitation fails, surgery appears to be a viable therapeutic option for restoring joint function by targeting and correcting the mechanical deficiency of the injured ankle–ligament complex [5]. The current scientific literature supports the use of different surgical strategies for treating ankle instability, ranging from anatomical repair of the native ligamentous complex to the use of different graft reconstruction techniques, performed via open surgery, minimally invasive and arthroscopic techniques [13, 22, 37]. Unfortunately, there is poor agreement on the surgical standard of care for CAI, and guidelines for determining the surgeon’s choice are still lacking [23, 41].

To compare and select appropriate surgical options for treating CAI properly, a combination of reliable tools, including both patient-related and clinician-generated parameters, must be considered [35]. Patient-reported outcome measures (PROs) are recognized modalities for accounting for the patient’s perspective on his/her current condition. This subjective view is of primary importance for the evaluation of any given treatment and should also ideally contribute in a positive manner to the clinical decision-making process. The ability of a PRO to produce clinically meaningful data is embodied in the multifaceted concept of validity, which can generally be defined as the ability of the instrument to measure the construct it purports to measure [6]. However, a critical analysis of the literature reveals that validity is an issue for current PROs used to assess CAI surgical outcomes [7, 20], raising some concerns about the quality of the evidence supporting clinical practice in patients with ankle instability.

The primary aim of this research was to address this knowledge gap by developing a new patient-reported outcome tool, following methodological guidelines, specifically designed to assess ankle instability and changes following therapeutic interventions. This study protocol describes the process of developing and validating a new tool to evaluate ankle instability, the Ankle Instability Treatment Index (AITI).

Why a new scale?

The best available evidence about the clinimetric properties of PROs in the specific CAI population suggests the use of the Foot and Ankle Ability Measure (FAAM), the Foot and Ankle Outcome Score (FAOS), and the Karlsson score as the most appropriate PROs for evaluating surgical outcomes in CAI patients [7, 12, 16]. The FAAM and the FAOS were originally conceived as region-specific scores to evaluate functional limitations associated with a variety of foot and ankle problems [21, 31]. Only retrospective evidence of validation has been obtained for patients suffering from ankle instability [3, 11, 30]. Both PROs thus do not specifically assess symptoms of ankle joint instability, which raises concerns about their ability to tap an essential disease-specific feature representing a primary target of any ankle stabilization procedure [40].

The Karlsson score was developed in 1991 to assess joint function after treatment for lateral ankle ligament injuries [16]. Since its inception, the scale has served as a useful tool in research dealing with the treatment of ankle instability, as evidenced by the frequent use of the scale to report the results of CAI surgery [34]. However, a systematic review published in 2007 on the available PROs in foot and ankle research area highlighted that the scale lacked evidence on important aspects of validity, such as content validity, reliability and responsiveness [20]. Since this observation, to the best of the authors’ knowledge, there has been no further analysis of the scale’s validity.

On the basis of these observations, the authors believe that the development of a new PRO for the evaluation of CAI surgical outcome, following the most recent guidelines on PRO properties, is justified by the current state of knowledge.

What should the scale measure?

A focus group consisting of all the authors of this publication (Dr. Pietro Spennacchio, Professor Jon Karlsson, Professor Romain Seil, Dr. Caroline Mouton and Dr. Eric Hamrin Senorski) with recognized expertise and previous publications in the field of ankle instability and outcome tools met initially to discuss the purpose and basic concepts of the new scale. The experts agreed that the main purpose of the project would be to develop an evaluative tool capable of assessing, through direct patient feedback, the symptomatic state of the CAI subject as well as its modification with treatment, according to what is most important to the patient.

Methods

The described development procedure adheres to the minimum requirements of validity and reliability as set forth by the latest version of the COnsensus based Standards for the selection of Health Status Measurements INstruments [COSMIN] [24]. The process of developing the new rating scale is shown in Fig. 1. It is a multistage process that involves iteratively and interactively, experts and patients in various qualitative and quantitative stages of development to produce a clinically meaningful scale [4]. To ensure the development of an instrument with high content validity, the process begins with a qualitative research phase aimed at exploring the subjective feelings and formulations of CAI subjects through individual face‒to-face interviews. The qualitative part of the research belongs to the “phenomenology” design type and can be related to the following question: “What do people with chronic ankle instability experience? “, with the aim of allowing participants to provide an insightful perspective on their subjective experience of living with ankle instability [18, 33]. The subjective feedback from the CAI subjects will then be used to support the definition of the construct to be assessed by the new scale.

Fig. 1

Flow diagram showing the multiphase process of AITI development. AITI: Ankle Instability Treatment Scale. CAI: Chronic ankle instability. PROs: patient-reported outcomes

Participants and recruitment

The inclusion criteria for participation in the development and validation of the new score are detailed in Table 1. The clinical diagnosis of chronic ankle instability reported in this study is consistent with the Position Statement on Selection Criteria for CAI subjects in Research defined by the International Ankle Consortium [12]. Recruitment will be conducted in a single center by a member of the focus group, who is an experienced foot and ankle surgeon (PS). In line with the stated phenomenological qualitative study design, sampling will be carried out via a criterion sampling strategy, with the most prominent criterion being the participant’s experience of the phenomenon of ankle instability, as supported by the diagnostic criteria outlined in Table 1 [18].

Table 1 Inclusion criteria

Patient interviews and data collection

Written informed consent will be obtained from all participants before the face-to-face interviews begin. A preliminary list of clinical features of ankle instability, derived from the experience of the developers and the content of PROs commonly used in the research dealing with ankle instability, will be defined to provide prior theoretical knowledge that will serve as a testing ground for the information emerging from the interviews. The interviews follow a framework of open-ended questions designed to encourage discussion of the patient’s subjective experience of the different dimensions of the pathology, as well as the change expected from a treatment designed to improve their current disease (Table 2). The interviewer will take special care to avoid any specific guidance or influence on the answers, to allow the participant to express his/her own feelings, perceptions and thoughts, using his/her own words as freely as possible.

Table 2 Qualitative research interview guide

The qualitative interviews will be conducted, transcribed verbatim and progressively coded by one researcher (PS). The raw data will be analyzed repeatedly from the first interviews onwards via an inductive coding scheme [8]. The aim is to define data with labels that will allow them to be grouped into preliminary categories that will allow the progressive coding of all the content collected during the interviews. The emerging categories will be analyzed for similarities in content and finally grouped into higher categories to establish a preliminary framework of the phenomenon of ankle instability, which comprises the different dimensions of the condition experienced by CAI patients [18]. The emerging categories and their content will be reported to the focus group. Any missing points suggested by comparisons with existing knowledge and the developers’ experience in treating ankle instability will be explored further with additional questions in subsequent interviews to iteratively configure the conceptual framework of ankle instability of the new scale.

The interviews will continue until saturation is reached, defined as the point at which no additional codes or insights emerge in three consecutive interviews, confirming clear data redundancy. On the basis of practical guidelines and estimates from previous qualitative phenomenological studies, a minimum of 10 face‒to‒face interviews are expected [24, 32].

Item generation, scale refinement and content validity

The conceptual framework developed will be used to design the domains and items of the new scale in its preliminary version. The information from the previous interviews will be used to generate relevant items, paying particular attention to the wording spontaneously evoked by the patient to ensure clarity and the patient-reported nature of the instrument.

The preliminary scale will be tested through a new round of face‒to‒face interviews with a minimum of 30 new participants not involved in the previous qualitative interviews who meet the same inclusion criteria, as described in Table 1 [24]. The purpose of the interviews will be to confirm the clarity of each instruction, item and response option. In the case of any unclear item or wording, the participant will be asked to explain his uncertainties and to suggest modifications that are able to improve the clarity of the question. Any possible missed aspects of the ankle instability construct will be further investigated through dedicated probing to explore the patient’s perspective on the content coverage of the scale. During the interview, a quantitative assessment of the content relevance of the scale will be carried out to confirm the instrument’s ability to analyze what matters to patients diagnosed with CAI [24, 28]. The respondents will be asked to rate the relevance of the items on a 4-point scale to calculate the content validity ratio for the item’s relevance and appropriateness of the scaling options [19].

The relevance of the items and the comprehensiveness of the instrument will be further investigated from the perspective of professionals (Orthopaedics and Physiotherapists) )with established experience in the treatment of ankle instability outside the development team. The AITI with a dedicated rating form will be emailed to these professionals, and they will be asked to rate the relevance of each item to the construct of ankle instability. The raters will also be asked to comment on whether any aspects of the construct of instability have been omitted.

How does the instrument work?

After content validity has been examined, the AITI will be subjected to an analysis of different measurement properties, as outlined below.

Construct validity

The construct validity of the AITI will be examined by defining its internal consistency, which is the extent to which the scale items are correlated with each other, thus measuring the same construct and supporting the derivation of a composite score from the sum of the items [38]. The cohort size of the subjects required to adequately determine the construct validity will be further defined when preliminary data will be available on a sample of 20 CAI patients to ensure statistical power for each analysis.

The correlation between items is defined by calculating the Cronbach’s α. Internal consistency between the items between 0.70 and 0.95 is considered acceptable [26]. To ensure a clear interpretation of the internal consistency statistics, the dimensionality of the scale will be tested with a confirmatory factor analysis [25]. The number of items making up the scale will also determine the appropriate recruitment size for internal consistency analysis, with a sample size of at least six times the number of items retained [24].

The construct validity of the AITI will be further explored by testing the hypothesis of an expected relationship with 2 scores commonly selected by researchers to assess CAI surgical outcomes in a minimum of 50 CAI patients [24]: the Karlsson scale [16] and the FAAM sports subscale [3]. The available evidence for the validation of the comparator instruments in CAI population subjects supports an expected relationship in the midrange of 0.4–0.8, as defined by the calculation of Pearson’s product‒moment correlation coefficients (parametric data) or Spearman’s r (rank correlation) coefficients (nonparametric correlation) [36].

Reliability

In addition to the definition of internal consistency described above, the reliability of the AITI will be further investigated by determining test reproducibility and measurement error in a sample size of CAI participants, which will be further defined once preliminary data are available with the new instrument. In accordance with the COSMIN guidelines, a minimum of 50 CAI subjects will be included in this analysis [24]. Reproducibility (test‒retest reliability) is the extent to which repeated measurements in stable individuals yield similar responses [38]. Patients participating in this step of validation will complete the new outcome scale twice, with a 10–14-day interval between the two administrations. In line with the definition of a PRO as information that comes directly from patients without interpretation by a clinician [27], the questionnaire will be administered in a strict self-administered mode without external support, which may introduce bias related to caregiver interpretation.

Evaluation of the test–retest reliability of the scale will be performed by calculating the intraclass correlation coefficient (ICC-agreement) with 95% confidence intervals (CI) [10]. On the basis of the ICC values, the standard error of measurement (SEM) and the minimal detectable change (MDC) will be calculated.

Responsiveness

Responsiveness is defined as the ability of a questionnaire to detect clinically important changes over time, even if these changes are small [36]. This is a fundamental property for any instrument purporting to evaluate the effect of a therapeutic intervention (evaluative instrument). The instrument responsiveness will be the last property to be analyzed, only after all the facets of validity outlined above have finally been proven to be at least as adequate [24]. A new group of minimum 30 CAI patients [24] will be analyzed using the instrument before and after an ankle stabilization procedure at a minimum follow-up of 1 year, a time point that is expected to show a modification of the preoperative patient’s health state. The effect size (ES) and the standardized response mean (SRM) will be determined as indicators of the ability of the new instrument to detect real changes [36].

Discussion

The most important direct patient perspective on a given treatment, captured through valid and reliable PROs, is considered an essential outcome for generating the data necessary to incorporate effective and meaningful treatment strategies into clinical practice [29]. The authors noted that the existing evidence on CAI surgical outcomes is mainly based on PROs with limited evidence of validity, which casts doubt on the consistency and reliability of the data supporting current treatment algorithms [20, 34].

This study protocol describes the process of developing and validating a new disease-specific patient-reported tool for the evaluation of ankle instability treatment, the AITI. The focus on patient input in defining scale content and adherence to the latest consensus-based standards for PRO validity and reliability represent the strategy for developing an instrument with appropriate measurement properties in CAI patients. The authors believe that this process is a necessary step in the search for scientifically sound data ensuring a reliable, evidence-based standard of care for patients suffering from ankle instability.