The Core Outcome set for Head, nEck and REspiratory disease in MPS II (COHERE) study follows the methodological principles set out from the recommendations developed by the COMET initiative [20]. Methods included a modified electronic Delphi (e-Delphi) process to achieve consensus of opinion among healthcare users and professionals working in the field internationally. Several challenges were faced during the development of this COS, such as limited healthcare providers with sufficient experience in the field, and limited engagement from healthcare users. As such, we had to modify the COS methodology in real-time. Therefore, a novel approach to developing a COS in a rare disease has been implemented. An overview of the methodological process adapted by the COHERE group can be seen in Fig. 1. The COHERE group therefore suggest a modified methodological process when determining a COS for a rare disease.

Fig. 1

Overview of the methodology adopted by the COHERE study group

Outcome identification

To develop the ‘long list’ of potentially important candidate outcomes, a preceding systematic review of the literature and clinical trials registry search was conducted [25]. Additionally, themes identified by qualitative research performed by Soni-Jaiswal were added [24]. From these sources, 46 outcomes were identified and grouped according to an existing outcome taxonomy [26]. Each of the 46 candidate outcomes was accompanied by a plain language definition (Appendix 1). Outcomes and definitions were reviewed and approved by the members of the study management group (SMG) and were presented to all the stakeholder representatives.

Stakeholder participants

The COHERE study aimed to consider the views of patients, families and/or carers, healthcare professionals, and clinical researchers working in the field. Healthcare professionals involved in the care of patients with MPS II were invited to participate, which included otolaryngologists, respiratory physicians, clinical geneticists, metabolic physicians, specialist nurses, physiotherapists, general paediatricians, and audiologists. Inclusion and exclusion criteria are below.

Inclusion criteria

Healthcare users:

Parents and/or carers:

Healthcare professionals:

Exclusion criteria

Healthcare users:

Parents and/or cares:

Healthcare professionals:

Recruitment

Recruitment was facilitated by professional networks and the MPS Society [27], reaching out to international MPS societies to advertise the study. Although there is no required sample size for an e-Delphi survey [28, 29]; the aim was to recruit at least 40 participants in total for the e-Delphi survey. One of the key-deciding factors for participant recruitment is that the participant panel membership should adequately represent corresponding stakeholder groups [20].

Purposive sampling methods were used to recruit healthcare professionals who were experts in metabolic disorders. Clinicians received group electronic advertisements via professional groups and/or personal contact from the study advisory group (SAG). Healthcare users received a group electronic advertisement with the study information through a trusted patient group (MPS Society, MetabERN [30]). Those who decided to take part were directed to the e-Delphi webpage which included detailed study participant information. If the participant agreed to take part, a link was provided for the online DelphiManager, a software maintained by the COMET initiative (https://www.comet-initiative.org/delphimanager/). The participants agreed to a series of consent statements prior to being able to proceed to the survey. Demographic information was also recorded, dependent upon the stakeholder group of the participant (Appendix 2).

e-Delphi survey

Delphi surveys have been used extensively for the development of COS [31]. To prioritise the outcomes to be included in the COS internationally, healthcare users and professionals were invited to take part in two modified e-Delphi surveys. Participants were not offered payment or vouchers for their time to take part in the e-Delphi.

e-Delphi survey—Round 1

Participants were presented with a choice of two e-Delphi surveys: for the age group 0–11 years old and another for 12–25 years old. Participants were allowed to complete both surveys if they considered themselves to have appropriate experience. Participants were asked to score each outcome domain on a 9-point scale proposed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) group [32]. Scores of 1 to 3 signified an outcome of ‘limited importance’, a score of 4 to 6 signified ‘important but not critical’ outcome, and a score of 7 to 9 signified a ‘critical’ outcome. Participants were also given an ‘unable to score’ option and could comment on any aspects of the scoring or outcome domains using open-text boxes. Round 1 participants were also able to suggest additional outcomes that they considered important. Suggested outcomes deemed to represent a new outcome domain by the SMG and discussed with the SAG were added to the list for consideration in round 2. Round 1 was open for approximately 6 months, with analysis of the responses lasting approximately one month prior to opening round 2 for voting.

e-Delphi survey—Round 2

Due to low level of engagement from parents/carers and patients, our methodology had to be modified. Therefore, round 2 was opened to healthcare providers only. Attempts were made to address the lack of participation from healthcare users by employing comprehensive support from an established information and news resource, the MPS Society. Both the MPS Society and the investigative team believed it was crucially important to include the perspectives of those with ‘lived experience’ in the final COS, as per established recommendations [33]. Therefore, perspectives of healthcare users would be gathered during the consensus meeting stage of the study.

All items, including additional outcome domains suggested by participants in round 1, were carried forward for consideration in round 2. Descriptive statistics were used to summarise the scores from round 1 and presented to round 2 participants. Participants could see the results of their individual score for each outcome in addition to the average score of each stakeholder group. The rationale for this approach was that it may improve consensus between the stakeholder groups [34]. Round 2 was opened for approximately 4 months, with response analysis lasting up to one month.

Missing responses

If a participant did not complete a subsequent round of the Delphi survey, their scores from previous rounds were counted as valid and retained in the study. Similarly, if a participant failed to score a specific item during a survey round, the answers to other items were held as valid and retained as long as the participant scored over 50% of all of the outcomes. Participants that failed to score over 50% of outcomes in round 1 were excluded from analysis and not contacted for round 2. This criterion has been successfully utilised by other COS development teams and ensured that participants contributing to the decision-making were fully engaged in the process and had sufficient expertise [28, 35].

Analysis of the e-Delphi surveys

Scores from round 2 were analysed using descriptive statistics and outcomes were grouped into the following categories, as per COMET initiative recommendations [21]:

1.

Consensus reached for inclusion in COS: outcomes scored as ‘critically important’ by greater than 70% of participants and ‘unimportant’ by less than 15% of round 2 participants,

2.

Consensus reached for exclusion from COS: outcomes scored as ‘critically important’ by less than 50% of round 2 participants, and

3.

Consensus not reached: any other combination of scores.

Outcomes that were ‘voted out’ were not carried forward and not included in the final COS (Fig. 1). The rationale for these categories was that for an outcome domain to be included in the COS, it requires agreement by the majority regarding the critical importance of the outcome, with only a small minority considering it to have little importance.

Consensus meetings

Integrating the opinions of experts by experience (i.e., healthcare users) is fundamental in developing a COS in a rare disease. Making COS development meaningful and accessible for healthcare users involves them having a genuine say in the development process [33]. The COHERE team were faced with lack of engagement from healthcare users during the e-Delphi stages. The challenge in securing meaningful input from healthcare users on an international scale was addressed by capturing the thoughts of healthcare users during the consensus meetings.

With the aid of the MPS Society, three virtual meetings were planned for differing age groups and hosted over Zoom [36]. All international MPS Societies and the Muenzer MPS Center [37] were contacted and asked to broadcast advertisement for the consensus meetings in an aid to boost international collaboration. The participant groups incorporated adults over 18 years of age (Group 1), teenagers aged 12–17 years old (Group 2), and children younger than 11 years old (Group 3). Parents or carers of the participants were invited to contribute for groups 2 and 3 also, and, if needed, in group 1.

In advance of the consensus meetings, participants revisited the outcomes and selected the three they deemed most important from the ‘voted in’ or ‘consensus not reached’ categories following the e-Delphi surveys (Fig. 1). This approach was successfully utilised in the development of a COS for single-sided deafness [28]. The identified outcomes were discussed during the corresponding consensus meeting to determine their inclusion or exclusion from the COS.

The consensus meetings provided opportunity for semi-structured discussions lasting approximately two hours. Discussions amongst participants were encouraged based on personal opinion and experience. Anonymised voting was conducted using the Zoom polling function to pre-determined questions. Results were presented to participants using histograms generated by Zoom. All consensus meeting discussions were recorded for analysis, recordings were deleted once this was completed.

‘Onion skin’ development

OMERACT (Outcome Measures in Rheumatology) principles were used when determining the classification of the final outcomes included in the agreed COS [38]. Hence, outcomes ‘voted in’ during the e-Delphi were included in the final COS. Likewise, outcomes identified for inclusion or those undecided by the consensus meetings were also included in the final COS. This methodology encompasses the perspectives of both healthcare professionals and healthcare users with expertise in MPS II by profession or experience. Following OMERACT principles, the selected outcomes were categorised as (1) mandatory outcomes; (2) mandatory outcomes for specific circumstances; (3) important but optional outcomes; and (4) research agenda outcomes. We acknowledge that the group of children and young people with MPS II is diverse and may have varying priorities depending on age, life stage, and health status, therefore a COS would be determined for each of the age groups separately.