This study will engage patients with MS and their family caregivers throughout MCDA to assess the values of DMTs. The engagement approaches will follow the recommendations from the patient engagement best practices resource document [39], the PCORI Engagement Rubric [40], and the PCORI Engagement Awards Evaluation Reporting Tool [41]. The project also follows two ISPOR MCDA Emerging Good Practices Task Force Reports to design and complete patient-centered value assessment [25, 26]. The study proposal will be submitted for approval from Auburn University and the University of Alabama at Birmingham Institutional Review Boards.

2.1 Study Samples

This project will mainly focus on the population impacted by relapsing-remitting MS (RRMS), the most prevalent MS phenotype [42]. Specifically, we will include patients living with RRMS and family caregivers with experience caring for patients with RRMS. Patients and their family caregivers living in Alabama and Mississippi from the patient-led MS research community established in our previous PCORI-funded engagement project [43] will be invited to join this study. They were trained and had experiences to engage effectively in patient-centered outcomes research/comparative effectiveness research (PCOR/CER). We will recruit additional patients and their family caregivers from Louisiana. For patients and their family caregivers from Louisiana, we will provide additional intensive training workshops (i.e., other than the workshop mentioned later) to ensure they can also engage in patient-centered research. These patients are aged ≥ 18 years and diagnosed with RRMS. Similarly, the family caregivers are ≥ 18 years of age and are affected by a family member with RRMS. At least two-thirds of participating patients are expected to be African American or people with disadvantaged SDOH, e.g., low incomes (less than median household income in each state) and education backgrounds (lower than a college degree). Generally, the appropriate sample size in MCDA is arbitrary. A recent systematic review indicated that the number of stakeholders who participated in scoring alternatives in previous MCDA studies varied widely [33]. The median numbers of individuals were 9, 10, and 16 for regulatory decisions, clinical decision making, and priority-setting studies, respectively. This project aims to include a total of 20 patients and 20 family caregivers across the three states.

2.2 Study Design

This study will be a cross-sectional study. Figure 1 shows the overarching workflow of this study. The study team includes a patient partner from the Alabama-Mississippi-Louisiana Chapter of the National Multiple Sclerosis Society (NMSS), healthcare provider partners (i.e., a neurologist, a nurse practitioner who specializes in MS, and also a qualitative researcher), and a value assessment researcher who is a pharmacist and has MCDA experience. The study team co-created an engagement plan when developing this protocol. The plan describes how patients and other stakeholders should be engaged throughout the study, including planning, conducting, and disseminating the study results.

Fig. 1

Overarching workflow and participants. The project team will compensate patients and family caregivers for their time, based on the rate suggested by the advisory committee throughout the project. MCDA multicriteria decision analysis, NMSS National Multiple Sclerosis Society, AL Alabama, MS Mississippi, LA Louisiana

The study will also establish an advisory committee to guide the project and ensure high patient and family member engagement. The committee comprises three patient representatives from the MS patient-centered research community at the national level (iConquerMSTM), a healthcare provider, two healthcare payers, and an industry representative. We intend to include more patient representatives than other types of representatives to ensure a high level of patient engagement in this study. The study team will meet with the Advisory Committee every other month to present the study progressions and plans and ask for their suggestions. The study team will address or respond to the suggestions accordingly. Additionally, the study team will reinforce six PCORI Engagement Principles, including reciprocal relationships, co-learning, partnerships, transparency, honesty, and trust, when we work together and with the Advisory Committee throughout the project. This study will compensate committee members for their time, based on agreeable rates. Patients and family caregivers will also be compensated for their time, based on the rate suggested by our advisory committee.

Primarily, this study is composed of two major phases, including (1) capacity building for value assessment focusing on MCDA, and (2) value assessment of DMTs for MS using MCDA.

2.2.1 Phase 1: Capacity Building for Value Assessment Focusing on Multicriteria Decision Analysis (MCDA)

The study team will use various resources, e.g., ISPOR value assessment frameworks [4], NHC value classroom [44], and ISPOR MCDA emerging good practices [25], to develop training materials, which will include multiple value assessment topics (see examples in Table 1). MCDA for healthcare decision making will be emphasized. The study team will develop the learning outcomes to ensure that the training materials will be for the introductory level of value assessment focusing on MCDA, which is the goal of the training workshops. After we develop the training materials, we will seek inputs from the advisory committee for two rounds. For the first round, we will ask the advisory committee to review the materials independently. Each committee member will be asked to comment on the content, appropriate language used in these materials, effective delivery format, and engagement needs. The study team will consolidate the comments and use them to adjust the materials. For the second round, we will share the adjusted materials with the committee again. We will conduct a group discussion with the advisory committee during the committee’s meetings. The project team will collect the comments from the discussion to improve and finalize the training materials.

Table 1 Examples of training topics

We will use the training materials to conduct six workshops on six different days in six months. The workshops will allow us to interact with the patients and their family caregivers, and they will be familiarized with the core concepts of value assessment. We mainly aim to introduce the overarching goal and terminology used in value assessment practices instead of training them as experts. We expect the workshops will help them recognize the importance of value assessment and understand the critical roles they can play. The format of these workshops will be similar to our previous PCORI-funded project workshops that were implemented successfully with this community [43]. For instance, we will conduct these workshops virtually. We will loan a tablet, e.g., GrandPad®, with internet access for people without a capable electronic device. We will ask the patients and family caregivers to review some materials, including short readings or videos, before they come to each workshop. Each workshop will last 2 h to prevent any burnout, since one of the major MS symptoms is fatigue. We will use the experts from the study team to deliver the workshop materials and train patients with MS and their family caregivers. We will enlist experts for specific topics and work with them to plan training workshops if necessary. We will use plain language during the workshops. We will provide many examples, including the Deep South context, which includes health disparities among African American patients with MS and patients with disadvantaged SDOH [45]. We will use a learner-centered teaching style, which actively engages participants through various activities, e.g., discussion and brainstorming. At the end of every workshop, we will evaluate their learning outcomes and ask them to evaluate the workshops. This evaluation will include reflection, an audience response system, and case discussion to field questions during the workshops.

2.2.2 Phase 2: Value Assessment of Disease-Modifying Therapies (DMTs) for Multiple Sclerosis (MS) Using MCDA

Primarily, this phase comprises three major steps: (1) instrument development (defining the decision problem, selecting and structuring criteria, measuring performance); (2) data collection and analysis (scoring and weighing criteria, calculating aggregate scores, testing uncertainty); and (3) interpretation of findings.

2.2.2.1 Instrument Development

The instrument development includes defining the decision problem, selecting and structuring criteria, and measuring performance. While describing the following processes for the patients, we will repeat these processes for the family caregivers.

Defining the decision problem: The scope of the problem will center on patient perspectives on the values of DMTs approved by the US FDA for MS. These values can be used to guide either reimbursement decisions or shared decision making between patients and healthcare providers. We will conduct a 2-hour virtual meeting to engage MS patients and seek their input in refining the project’s scope and the decision problem. We will adjust the scope and decision problem based on patient feedback accordingly.

Selecting and structuring criteria: We will use the framework for developing disease-specific patient-centered core impact sets (PC-CIS) developed by the NHC multistakeholder advisory group to guide criteria selection [46]. We will review the literature, including patient preference and disparity studies [47, 48], to generate a list of criteria and their performance ranges. These criteria will guide our semi-structured focus group interviews with MS patients. Specifically, the list of criteria will help the study team develop interview questions to guide the group discussions. Essentially, the interview questions will be open-ended, e.g., “What value components for MS treatments are important to you?” However, we may use close-ended questions, e.g., “What do you think about ‘X (a potential criterion)’ in terms of the DMT treatments?”, to confirm some criteria that were identified from the literature but not mentioned while using the open-ended questions. We anticipate 2 h for each interview. Each interview will include approximately six patients. The interview data will be transcribed and analyzed along with the findings from the literature review. We will conduct the interviews until the data become saturated. At the end of the interviews, we will ask patients to rank the importance of the identified criteria. After gathering all criteria along with ranking from patients, the project team will ask the Advisory Committee, which also includes patient representatives and other stakeholders, to review and choose the criteria based on their relevance and simplicity of assessment, understandability for patients, and applicability in the data collection process. We anticipate that these criteria will include not only the benefits and risks of DMTs but also some novel value elements, e.g., equity. We will structure these criteria in a visual manner to show the composition of the overall value of DMTs.

Measuring performance: Primarily, we will use evidence from systematic literature reviews and network meta-analyses to measure the performance of the criteria. If no evidence exists for any criterion, we will ask clinical experts from our study team and the Advisory Committee to provide their opinions. We will construct a performance matrix, showing the performance of DMTs against the criteria. This matrix will include estimates of average performance, variances of these estimates, and the data sources. At the end of this step, we will conduct a 2-hour virtual meeting to share the criteria and their performance matrix and obtain the patients’ input. We will adjust the criteria and their performance matrix based on their input.

2.2.2.2 Data Collection and Analysis

Figure 2 shows the data collection and analysis, which include scoring alternatives and weighing criteria, calculating aggregate scores, and testing uncertainty. We will conduct two in-person meetings for this step—one for the patients and another for the family caregivers. Similar to the previous step, we will describe the following processes for the patients and repeat the processes for the family caregivers. There are several approaches for scoring and weighing in MCDA. These approaches include compositional and de-compositional methods [25, 26]. While the composition methods generate separate estimates of scores and weights, the de-compositional methods derive scores and weights simultaneously. No method is clearly superior [26]. We propose using the compositional method in this study because it will likely be less complex for the patients and family caregivers; however, the Advisory Committee will be engaged in this decision. All analyses will be performed using Microsoft Excel® (Microsoft Corporation, Redmond, WA, USA).

Fig. 2

Data collection and analysis process. DMT disease-modifying therapy

Scoring treatment performance: We will capture patients’ priorities or preferences for changes in performance within each criterion when scoring DMT alternatives. To incorporate performance estimates into MCDA, we will use partial value functions to convert them into scores, defined on a scale of 0–100 points. Therefore, all criteria will be measured on comparable scales. We will also consider whether highor low performance is better when establishing the partial value functions. To choose the partial value function, we will ask whether each patient's value performance is linear between the two extreme ends (0 and 100). If not, we will use a bisection and difference method to develop a nonlinear partial value function. That is, we will ask each patient to identify the performance level of each criterion that is worth 50 points on the scale and then identify the score on the 0–100 scale for the midpoint on the range of performance [26]. We will repeat this process to define the shape of the value function. We will share all value functions with every patient and discuss the differences. After the discussion, each patient can adjust their functions, but we will not force consensus. We will use each patient’s value function to score DMT performance on each criterion.

Weighing criteria: There are various weighting methods with different resource requirements, the chance of bias, and complexity [49]. We proposed using a swing weighting method. The swing weight method is a relatively simple approach and requires a low level of resources. It was also used by the European Medicines Agency (EMA). For the swing weighting method, we will apply a hierarchical technique used in a previous study to explicitly account for the positive and negative impacts of the criteria [50]. We will first categorize all criteria, e.g., benefits and risks. Next, we generate weights within each category of criteria and then between categories by comparing the highest weighted criterion from each category. To do so, we will present the performance range (i.e., the swing) of each criterion to the patients and ask them individually which range is the most important to them. We will assign a weight of 100 points to that criterion and ask the patients to assign 0–100 points to other criteria to reflect the relative importance compared with the most important criterion. To validate these weights, we will ask questions such as “Would a change of a to a’ on criterion A be valued x times as much as a change from c to c’ on criterion C?” We will share individual weights and discuss the differences. After the discussion, each patient can update their proposed weights, but we will not force consensus. We will normalize the weights of positive categories (e.g., benefits) to sum to one and adjust the weights of negative categories (e.g., risks) to maintain the relative importance with the criteria of the positive categories.

Calculating aggregate scores: We will use an additive model assuming the selected criteria are independent since patients will be asked to consider redundancy or overlapping during the selection of related criteria in instrument development, and they will be asked to score treatment performance independently. We will estimate each DMT’s weighted score, which is the sum of the products of each criterion’s (normalized) mean weight and mean score. We will subtract the weighted sum of the negative criterion (e.g., risks) scores from the weighted sum of the positive criterion (e.g., benefits) scores.

Testing uncertainty: We will test for parameter uncertainty, e.g., uncertainty in the performance ranges of the criteria, by conducting both one-way and probabilistic sensitivity analyses. We will vary one value of the criteria performance at a time for the one-way sensitivity analysis. We will also perform 5000 iterations for the probabilistic sensitivity analysis. In each iteration, we will randomly select one of the patients’ value functions and weights. We will conduct scenario analyses to test for structural uncertainty (e.g., selected criteria) using different sets of criteria. We will also examine the heterogeneity in preferences among subgroups identified by our advisory committee.

2.2.2.3 Interpretation of Findings

We will present the findings of this project in tabular or graphical forms to our advisory committee and allow them to discuss how to interpret and disseminate the findings before we conduct a 2-hour virtual meeting to present and discuss the findings with the patients and family caregivers.