Establishing a group of experts to determine consensus prior opinion

A consensus meeting was held over two days which brought together four experts from the United Kingdom and five experts from across Europe. Invitations were initially sent out to 29 experts across the United Kingdom and Europe and both rheumatologists and ophthalmologists were invited. Experts needed to have had recent experience in treating JIA-associated uveitis, had no connection to the planned trial and be willing to take part in the expert elicitation meeting. Based on responses to invitations, all nine experts who volunteered were invited to take part in the face-to-face meeting. The purpose of the meeting was to assess expert consensus but not to confirm efficacy of secukinumab. This meeting was held before the trial began rather than at the conclusion of the first stage of the trial. The idea of using elicited information is to strengthen the trial results by augmenting them with expert opinion. Using data from the study to determine how much to strengthen would lead to potential double counting information (rather than augmenting with additional information) and should therefore be avoided.

Process of Establishing consensus prior opinion

The meeting took place in London, UK on the 10th and 11th of October 2023 and proceeded according to the agenda listed in Table 1. The first day focussed on describing the planned study, the study end points and a detailed review of the existing evidence around treatment options for uveitis associated with JIA. Results of available evidence and scientific rationale and pre-clinical evidence behind the use of IL-17 A inhibition in uveitis were also presented at the meeting Additionally, the statistical framework that would be used was introduced and the approach to elicitation discussed. Finally, a mock elicitation exercise using a fictitious example was held to ensure the process and the questions asked was fully understood.

Table 1 Activities comprising the consensus meeting and the time dedicated to each

The second day was devoted to the formal elicitation process, structured so that the opinions of individuals were established before attempting to reach a consensus. This structure was adopted to reduce the risk of experts being unduly influenced by overconfident group members or those with strong personalities. Three facilitators (TJ, GC, DR) with statistical training in Bayesian methodology were on hand throughout to facilitate. Each expert was asked to complete a structured questionnaire (see Supplementary materials). They then had a one-to-one meeting with a facilitator during which the answers provided were visualised (and potentially adjusted) to ensure they reflected the experts’ opinion. Neither the trial Chief Investigators (AVR, MWB) nor the trial lead ophthalmologists (CG, AD) took part in the questionnaires. After each expert had completed their one-to-one meeting the facilitators summarised answers and a structured discussion ensued, moderated by the trial Chief Investigators (AVR, MWB) and a statistical facilitator (TJ).

Approach for Establishing bayesian prior distributions

The goal of this elicitation exercise was to characterise the current understanding about the effectiveness of adalimumab and secukinumab and to construct informative Bayesian prior distributions to be combined with future trial data. In the following we followed the statistical methodology described in [21] and applied in [22].

Defining the quantities to be elicited

The elicitation focused on the planned primary endpoint of the study. This is defined as response to treatment as per SUN criteria as a 2-step decrease in the level of inflammation (anterior chamber cells) or decrease to zero between baseline (prior to trial treatment initiation) and after 12 weeks of treatment.

This endpoint was used in the individual elicitations and formed the basis for the consensus discussion. During the consensus discussion, which included both ophthalmology and rheumatology experts, however, the experts suggested that a slightly modified endpoint might result in a more meaningful trial. The proposal was to consider additionally a decrease to 0.5 (instead of 0) already as response. Rather than deciding about which endpoint the future stage of the trial should adopt, an ad-hoc decision was made to also obtain a consensus opinion on the modified endpoint as well as the current definition of response, to enable it to be used should investigators decide to do so.

Statistical model

Prior distributions were found for the response rate of adalimumab (ADA) and secukinumab (SEC), pADA and pSEC while the log odds ratio, defined as

$$\:\theta\:=log\left(\frac_\left(1-_\right)}_\left(1-_\right)}\right),$$

was used to compare the two treatments. Positive values of θ imply that SEC is superior to ADA and conversely for negative values.

Following [21] the parameters pADA, and θ were elicited directly from the experts. A beta distribution was used to model pADA while a normal distribution was used for θ. The prior distribution of pSEC was numerically derived from the distributions of pADA, and θ. This was done so that the pADA and θ could be treated as independent of each other.

Establishing expert opinion

We first asked each expert to provide initial responses to the questions which subsequently were discussed in a one-to-one meeting with a statistician. An R Shiny application [23] was used for visualizing the prior distributions implied by the initial responses and alterations were made as required until the resulting distributions adequately reflected the expert’s opinion.

These individual prior distributions were then presented to all experts to initiate a discussion of the opinions. The goal was to arrive at a consensus prior distribution via behavioural aggregation following the Sheffield Elicitation Framework [24]. The facilitators of the group discussion highlighted apparent discrepancies between distributions to stimulate discussion.

Should, against expectation, no consensus be achieved, the plan was to use mathematical aggregation of the individual experts’ distribution using equal weight for each.