Overview

This study will validate the TAPS-ESP in both interviewer- and self-administered formats, largely mirroring the methodology of the seminal English language validation study.

This project is motivated and informed by the principles of the Conceptual Framework for Advancing Health Disparities Research within the Health Care System described by Kilbourne et al. [46]. Briefly, this framework posits a nested structure of factors that perpetuate health disparities within the health services arena, including healthcare system factors (organization, financing, culture, etc.), within which patient and provider factors intersect at the clinical encounter. It is at this level that patient-provider communication and cultural competence are paramount. The availability of linguistically-accurate screening and brief assessment resources (especially if paired with clinical decision support) has the potential to improve provider competency in addressing substance use problems, while elevating patients’ receptivity to provider communication about behavior change, harm reduction, or referral. The framework further organizes disparities-focused services research into phases of (a) detection (identifying, measuring, and tracking disease in a defined population), (b) understanding (determinants and perpetuators of disparate service access and outcomes, including at the level of patients, the clinical encounter, and the health care system), and (c) reduction (translation and dissemination of evidence-based practices and policies). The TAPS-ESP project is focused on the detection phase but will also target the understanding phase by considering acceptability of the TAPS-ESP among clinic patients and perceived barriers and facilitators to its use by providers. In a broad sense, the TAPS-ESP effort seeks to directly address gaps in access to evidence-based substance use screening for a health disparity population by removing linguistic barriers and developing the evidence base with the target population.

Setting and recruitment

The TAPS-ESP validation study will be conducted at Baylor Scott and White Health (BSWH), the largest non-profit health system in Texas and one of the largest in the US. The BSWH system includes multiple primary care sites across the Dallas/Ft. Worth metroplex. Participants will be recruited from up to seven primary care sites of the Community Clinic network operated by BSWH. Depending on the clinic site, Hispanic patients make up 51%-80% of the patient population, and the majority are Spanish-dominant or Spanish-preferred. All research staff will be bilingual, native Spanish speakers. The providers in the clinic will deliver care in Spanish, some as native Spanish speakers, some with interpreters present.

We will use a multi-pronged recruitment strategy, including (a) direct invitations by research staff in the clinic waiting room, (b) direct invitations from telehealth visit schedules, (c) direct provider referrals, and (d) patient-initiated contact from advertisements posted in the clinic sites. Within these approaches, we will focus recruitment on the general primary care population, as well as targeting patients who are engaged with the behavioral health services provided within BSWH (where prevalence of substance use is likely to be higher).

Inclusion criteria will be: (1) age 18 or older; (2) BSWH patient; and (3) Spanish-language preferred (with the ability to read Spanish).

Exclusion criteria will be: (1) unable to provide informed consent (e.g., due to impairment or psychosis). All research activities (e.g., informed consent discussion and interview) will occur in Spanish. Participants will sign an IRB-approved Spanish language consent form.

Study procedures and informed consent

Procedures for the validation study involve a one-time research visit. All participants will complete the TAPS-ESP in both interviewer- and self-administered formats. However, to test for administration order effects, the order of the format will be determined at random, see Fig. 1 for SPIRIT flow diagram. Randomization addresses confounding by administration order, thereby allowing an unbiased comparison of self- and interviewer-administration formats (both of which are widely used in clinical care). This design was used in the original TAPS study and our prior work [42]. Randomization is automated, with minimal extra time burden.

Fig. 1

Recruitment and procedures flow for the validation study

After completing the TAPS-ESP in both formats, participants will complete a battery of self-report measures for validation purposes, as well as a brief satisfaction and acceptability questionnaire. Self-report measures will be interviewer administered only. Participants will receive $40 for completing the self-report portion of the validation study, which is expected to take about 1.5–2.0 h for participants who report polysubstance use. Because this is a validation study, participants will be informed at the outset that each measure must be asked in its entirety, and to expect some redundancy in the questions. Nevertheless, the assessment may be slightly shorter in duration for participants with limited substance use due to skip patterns that exist within each measure.

At the conclusion of the self-report measures, participants will be invited to provide an oral fluid cheek swab, which will undergo rapid assay testing for biomarkers of recent substance use. Participants will be asked to sign a separate consent for the oral fluid testing and will receive an extra $10 for providing a sample. This approach will be undertaken to ensure that self-report measures are assessed free of bias that may occur if participants know ahead of time that they will be tested. This will serve as a test of the veracity of the self-report information obtained from participants. We will track refusal to consent to the oral fluid test component and will analyze these data under different assumptions (e.g., completers only; refusals imputed as positive). Participants will also be asked about medications they are taking as prescribed in order to account for medicines that may be reactive with the tests.

Measures

All measures will be administered in Spanish-language translations. These measures were selected because they represent widely used comparative standards for which Spanish translations are available. Thus, for this reason and for consistency with the broader screening literature, our primary analysis will focus on validating the TAPS-ESP against the DSM-5 SUD diagnostic criteria. Other screening tools will be examined with respect to establishing concurrent validity. The measures are summarized in Table 1, and each is described below.

Table 1 Summary of primary and secondary substance use measures for TAPS-ESP validationPrimary criterion standard for validation

Modified World Mental Health Composite International Diagnostic Interview (CIDI): The CIDI is a comprehensive instrument used to assess mental health disorders based on criteria in the International Classification of Diseases (ICD-10) and the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV and DSM-5). Substances covered in the CIDI are alcohol, tobacco, and nine substance categories. As is the methodological standard in the field and as done in our prior research [42], we will administer a modified version of the CIDI consisting of the subset of items mapping to DSM-5 substance use disorder (SUD) criteria, asked for each endorsed substance class. We will examine categories of unhealthy use (1 DSM criterion) and SUD (2 + criteria), using the CIDI as the “gold standard” for identifying SUDs (i.e., patients at higher risk) [47, 48]. This is consistent with the methodology of the original English-language TAPS study and many other validation studies. The CIDI items mapped to the DSM-5 criteria for each substance will serve as the primary criterion standard.

Secondary self-report measures

Several secondary measures will be administered for the purposes of assessing the concurrent and convergent validity of the TAPS Tool. Many of these tools are limited to one substance (e.g., CAGE, FTDN), or assess for problems that are not substance-specific (e.g., DAST-10).

Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST): The ASSIST was developed for the World Health Organization to screen for alcohol, tobacco, and drug use in medical care settings. For each substance, the use dimensions include lifetime use, past 3-month use, urges or cravings to use, and adverse consequences from use, as well as concerns expressed by family or friends about use (lifetime, past 3 months), failed attempts to control, cut down, or stop use (lifetime, past 3 months), and drug injection (lifetime, past 3 months). The ASSIST provides substance-specific risk scores, 9 substance classes, with scores of 1–3 corresponding to low risk, 4–26 (10–26 for alcohol) to moderate risk, and 27 or higher to high risk [49]. As a WHO instrument, the ASSIST is available in many languages, including Spanish. A number of psychometric or validation studies of the Spanish language version of the ASSIST have been conducted [50].

Drug Abuse Screening Test (DAST-10): The DAST-10 is a ten-item yes/no screen for general drug use problems (not including alcohol). Each item is worth one point, and respondents are tiered into risk categories based on score, with more intensive assessment recommended at a score of 6 or higher. It has been widely used in the substance use field but does not distinguish between types of drugs used. A Spanish language version has been developed [39].

CAGE: The CAGE is a rapid alcoholism screening test comprised of four yes/no questions (Cut Down, Annoyed, Guilty, Eye-Opener) [51]. A Spanish language version was validated in both Spain and the US, where it performed well in identifying DSM-IV alcohol abuse or dependence among Latino/a primary care patients [38]. An affirmative response to any of the items was sensitive in identifying DSM-IV alcohol abuse or dependence.

Alcohol Use Disorders Identification Test (AUDIT): The AUDIT is a 10-item questionnaire which covers the domains of alcohol consumption, drinking behavior, and alcohol-related problems. It was developed from a six-country World Health Organization collaborative project as a screening instrument for hazardous and harmful alcohol consumption [52]. Responses to each of the 10 questions are scored according to a frequency rating of 0 (never) to 4 (daily), giving the entire questionnaire a possible score of 40. A score of 8 or more indicates harmful or hazardous alcohol use. A Spanish language version of the AUDIT was tested within the US in a small study, where only 51% of patients with alcohol use disorder were identified at the standard cut-point [38]. Thus, this widely used tool may not be optimal for Spanish-speaking populations in the US. We will administer the AUDIT to directly quantify the differences with the TAPS Tool in identifying alcohol use disorder.

Fagerstrom Test for Nicotine Dependence (FTND): The FTND is comprised of six questions scored on a point system, with total scoring summing between 0 and 10. Higher scores indicate heavier reliance on nicotine. The FTDN has been widely used in the tobacco field for decades [53]. It has been found to be internally consistent and an acceptable way to measure nicotine/tobacco dependency. The FTND has been translated into Spanish and validated. [54]

Biomarkers of substance use

Oral Fluid Tests As done in the original TAPS study [16], oral fluid assay testing will be used to detect recent use of substances including alcohol, nicotine, cannabis, cocaine, amphetamines, sedatives (benzodiazepines), and opioids (heroin/morphine metabolite, oxycodone, buprenorphine, methadone, and fentanyl). Participants will be asked about prescribed medications to account for cross-reactive metabolites. Although oral fluid only captures recent use (within the past few days for most substances), it is useful as an objective biomarker to validate self-report. The study will use the Orawell® 12-panel saliva test plus alcohol and the NicDetect oral fluid test for cotinine. According to the product inserts, detection cut-offs are as follows: amphetamines (25 ng/mL), methamphetamine (25 ng/ML), barbiturates (25 ng/mL), benzodiazepines (10 ng/ml), cocaine (20 ng/mL), morphine (10 ng/mL), THC (20 ng/mL), oxycodone (40 ng/mL), methadone (30 ng/mL), buprenorphine (10 ng/mL), fentanyl (10 ng/mL), tramadol (25 ng/mL), alcohol (0.02% BAC), and cotinine (30 ng/mL).

Process Measure

Satisfaction and Acceptability Questionnaire After completing the TAPS-ESP (and prior to completing the other measures), participants will be asked to complete a brief questionnaire to gauge feasibility and acceptability of the TAPS-ESP. This survey will mirror the feasibility/acceptability questions used in our prior work. Sample questions include “How much do you agree with the following statements?” (rated on a 5-point Likert-type scale; e.g., “not at all true” to “very true”): I would be willing to answer questions like these at my doctor’s office every year; I answered the questions as honestly as I could; I think my friends and family would answer these questions honestly at their doctor’s office; The questions were easy to understand; The touchscreen tablet was easy to use.” In addition, participants will be asked about their preferences and comfort with different ways in which TAPS-ESP information could be used clinically (e.g., automatic provider notification regarding patients’ screening scores), and preferences for intervention among participants who screen positive (e.g., provider-delivered vs. interactive computerized intervention).

Statistical analysis

Validation of the TAPS-ESP will follow the roadmap provided by the original English language TAPS study [16]. First, validation of the TAPS-ESP will be done for the broad substance categories in the TAPS-1 screening portion of the Tool (tobacco, alcohol, illicit drugs, non-medical use of prescription drugs). Second, we will conduct validation with the entire TAPS Tool for more specific substance classes (tobacco, alcohol, cannabis, cocaine, illicit amphetamines/methamphetamines, non-medical use of amphetamine-type stimulant medications, heroin/illicit opioids, non-medical use of opioid analgesics, and non-medical use of sedative medications). In addition, we will conduct analyses that combine logical categories (for example, any SUD other than tobacco; opioids inclusive of heroin/illicit opioids and non-medical use of opioid medications; all illicit drugs other than cannabis).

Analyses of TAPS-ESP performance

The TAPS-ESP scores will be validated against the CIDI as the primary diagnostic reference standard. Secondary screeners and assessments will be similarly examined from the standpoint of concurrent validity. Statistical analysis of TAPS-ESP screening test performance will employ a receiver operating characteristics (ROC) approach, including appraisal of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the curve (AUC). Optimal score cut-points will be confirmed using the Youden’s J statistic (unweighted and weighted prioritize sensitivity). Our target sensitivity and specificity will follow the established standards in screening test research, where test performance is considered acceptable, good, and excellent, at value thresholds of 0.7, 0.8, and 0.9, respectively. In addition, we will use the concordance correlation coefficient (CCC) [55], similar to a weighted κ statistic, to compare the agreement, accuracy, and precision associated with the screening success of the TAPS-ESP. For detecting the CIDI-derived DSM-5 substance use disorder diagnosis, we will examine TAPS-ESP cut-points for the 4-item screener as well as the brief assessment component in detecting levels of problem severity defined on the basis of DSM-5 criteria. As done in the original TAPS study, we will examine cut-points for detecting three different tiers of unhealthy substance use: Problem use (1 + criterion), SUD (2 + criteria), and moderate-to-severe SUD (4–11 criteria). Detection of SUD at any level will be the primary outcome of interest for each substance category. We will also examine performance of the TAPS-ESP in eliciting disclosure of substance use, irrespective of endorsing SUD-related problems.

Comparisons by administration format, sex, and age

We will conduct all analyses separately for (a) the interviewer-administered TAPS-ESP, and (b) the self-administered TAPS-ESP. In addition, we exploit the random-order design to directly compare the administration formats in their ability to elicit self-disclosure of substance use, and in their performance with respect to sensitivity, specificity, and ROC curves. We anticipate a fairly balanced representation of male and female participants, as well as adults across the lifespan, and thus will also examine TAPS-ESP performance based on participant sex and age. We will employ a combination of subgroup analyses and multivariable logistic regression models to comprehensively examine these factors. This approach replicates the rigorous analysis strategy of the English language TAPS study [16, 18, 35].

Satisfaction and acceptability

Data on satisfaction and acceptability will be examined descriptively, with target benchmarks of ≥ 4/5 for each rating. We will examine whether satisfaction and acceptability ratings differ by participants’ reported substance use behaviors using χ2 tests of independence.

Power

Our power analysis used the approach for minimum sample size determination for sensitivity and specificity analyses as recommended by Bujang and Adnan [56]. These authors computed minimal sample size requirements for screening and diagnostic validation studies under a range of scenarios regarding the prevalence of the disease in the clinical population. Although substance use problems as a whole are prevalent in primary care populations, prevalence is expected to be low for specific SUDs. For a screening study, particularly one with a low prevalence of disease, achieving high sensitivity (i.e., the ability of the test to accurately detect a true case of the condition; true positives) is paramount, and much lower sample sizes are needed for specificity. Thus, we present power calculations for sensitivity as recommended for screening studies, where the prevalence of the disease/condition is low, ranging from 2.5–20%, where power > 0.80 and the null hypothesis is 0.50. Our target sample size of N = 1000 exceeds the maximum sample size requirements for screening studies even at a low prevalence of 5%, even if the screening performs only in the “acceptable” range (Table 2). If, as expected, the TAPS-ESP performs in the “good” or “excellent” range (with sensitivities exceeding 0.80 or 0.90), a smaller sample size would be sufficient, even at very low prevalence of 2.5%. The approach detailed above is recommended for screening studies against a diagnostic standard. Nonetheless, we also computed power to detect differences in sensitivity between the TAPS-ESP and other screeners, using the Stata/SE 16 PSS command suite for differences in marginal proportions in dependent samples. We assumed high within-subject correlations and base sensitivity of 0.70 in the comparison screener. At 10% prevalence, power to detect a 10% improvement in sensitivity approached 0.90, but fell below 0.70 when prevalence was set at 5%. Power remained good (0.80) at prevalence of 5% to detect a slightly larger improvement in sensitivity of 15%.

Table 2 Minimum sample size for sensitivity in screening studies, with power > 0.80, α = 0.05, and H0 = 0.50Provider recruitment

Once the TAPS-ESP has completed validation, we will recruit 10 primary care providers to obtain perspectives on barriers, facilitators, and preferences regarding screening with the TAPS-ESP. Participants will be eligible if they are a licensed clinical provider at BSWH (e.g., physician, nurse practitioner, physician’s assistant, clinical social worker), who routinely treats adult patients. Providers will receive training on using the TAPS-ESP resources within the BSWH health information technology system,

Qualitative interviews

Providers will complete a face-to-face qualitative interview focused on screening and the TAPS-ESP. Interview questions will focus on factors related to providers’ perceived competencies, current screening practices and experiences, and how the TAPS-ESP could fit into the clinical workflow. Interviews will be audio recorded and transcribed for analysis. Data analysis will use a content analysis approach, with themes organized based on the tenets of Kilbourne et al.’s Conceptual Framework for Advancing Health Disparities Research within the Health Care System [46]. Two coders will analyze the data independently and meet to discuss and rectify discrepant interpretations, and ultimately reach consensus on key emerging themes.82 In addition, the narratives will be subjected to analysis by a third independent coder using an open coding strategy akin to the first steps in the grounded theory method [57], approaching the dataset without preconceived ideas about what themes and topic areas may be nested within the narrative. The third coder will then meet with the two other coders to discuss the extent of alignment between their interpretations of the data. This approach to triangulation will help to ensure that the qualitative data from providers are examined in a comprehensive way, from multiple vantage points.