Background As multiple strategies have emerged for managing treatment-resistant major depressive disorder, efficient identification of individuals at elevated risk for this outcome earlier in their illness course remains essential.

Method We extracted electronic health records data for all individuals with a diagnosis of major depressive disorder who received an index antidepressant prescription in the clinical networks of three geographically-distinct health systems – Mass General-Brigham (MGB), Vanderbilt University Medical Center (VUMC), and Geisinger Clinic (GC) – between April 1, 2004, and March 30, 2022. The primary outcome, treatment resistant depression, was defined as provision of electroconvulsive therapy, transcranial magnetic stimulation, vagus nerve stimulation, prescription of either ketamine or esketamine or monoamine oxidase inhibitors (MAOIs), or failed trials of more than two antidepressants. We applied L1-regularized regression to sociodemographic features, medications, and ICD10 diagnostic code counts to fit a model of treatment resistance in each of the three cohorts. For each, we then estimated generalizable model performance, aka external validity, across the other two cohorts. Model concordance was measured with Concordance Correlation Coefficients (CCCs) and random forest regression analyses were used to estimate importance of features predicting discordance.

Results Across sites, discrimination performance ranged from Area Under the Receiver Operating Characteristic curves (AUROCs) 0.58 – 0.64 on internal validation and 0.51 - 0.58 on external validation. Area Under the Precision-Recall curve (AUPRC) ranged from 0.1-0.13 on internal validation and averaged 0.07-0.13 in external validation on the same test sets held out at each site. On the same testing set, CCCs were 0.13 for the VUMC<-> MGB models, 0.18 for VUMC<->GC models, and 0.38 for MGB<-> GC models. These results indicate the MGB and GC models were better correlated, but none were well correlated. Important features predicting discordance were dominated primarily by age and secondarily coded sex.

Conclusion These linear models demonstrated consistent aggregate performance and discordant individual performance across three, disparate major health systems. The inclusion of large and heterogeneous samples suggest that further improvement may require incorporation of data types beyond those readily available in EHR. Close attention to performance by key subgroups is indicated to ensure models do not perform disparately or unfairly. Prospective studies to evaluate the extent to which clinical models might improve early identification and outcomes are warranted.

Dr. Perlis has received consulting fees from Alkermes, Circular Genomics, and Genomind. He holds equity in Circular Genomics. He serves as editor in chief of JAMA+ AI, and AI editor of JAMA Network Open. Dr. Walsh has received consulting fees from Newport Health, Humana, Prizam Healthcare. Dr. McCoy serves as a paid editor of Nature Digital Medicine. Dr. Ruderfer has served on advisory boards for Illumina and Alkermes and has received research funds unrelated to this work from PTC Therapeutics and Sanofi.

Dr. Perlis is supported by the National Institute of Mental Health (R01MH123804 and U01MH136059). Dr. Ruderfer and Dr. Walsh are supported by (R01MH121455 and R01MH11629).

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

* Vanderbilt University Medical Center Institutional Review Board, # 151156 and #151797, Approved * Mass General Brigham Human Research Committee, Approved * Geisinger Institutional Review Board, IRB #2020-0188, Approved

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Disclosures: Dr. Perlis has received consulting fees from Alkermes, Circular Genomics, and Genomind. He holds equity in Circular Genomics. He serves as editor in chief of JAMA+ AI, and AI editor of JAMA Network Open. Dr. Walsh has received consulting fees from Newport Health, Humana, Prizam Healthcare. Dr. McCoy serves as a paid editor of Nature Digital Medicine. Dr. Ruderfer has served on advisory boards for Illumina and Alkermes and has received research funds unrelated to this work from PTC Therapeutics and Sanofi.

Funding: Dr. Perlis is supported by the National Institute of Mental Health (R01MH123804 and U01MH136059). Dr. Ruderfer and Dr. Walsh are supported by (R01MH121455and R01MH11629).

Because of the clinical EHR nature of the study, datasets are not available for dissemination or sharing.