Prediction of individual treatment allocation between electroconvulsive therapy or ketamine using the Personalized Advantage Index

Introduction

Major depressive episodes (MDE) affect an estimated 21 million adults in the United States annually¹. Although numerous pharmaceutical, behavioral, and device treatments are available, only about one-third of patients achieve remission following standard first-line treatments² and after failing two or more antidepressant treatments, patients are classified as having treatment-resistant depression (TRD)³. The morbidity and mortality associated with this high rate of failed treatment response have heightened interest in the development of more personalized medicine strategies⁴ (e.g., precision therapeutics) that might prospectively allocate patients to treatments best suited for them, reduce reliance on trial-and-error strategies and increase the speed and magnitude of response. Numerous studies have investigated specific predictors or correlates of antidepressant treatment outcomes using neuroimaging^5,6,7,8,9, genetic^10,11,12, and clinical history measures^13,14,15. While identification of treatment-responsive biomarkers for a given treatment is highly valuable, studies focused on a single treatment modality do not provide clinicians with a direct means by which to differentially assign patients to an optimal treatment. To address this limitation, DeRubeis et al.¹⁶ introduced the Personalized Advantage Index (PAI). The PAI method is a means of identifying treatment-prescriptive variables (e.g., biomarkers, behavioral, demographic, or clinical data) that predict differential treatment outcomes across two or more treatments¹⁷. Applied to a set of two or more treatments, the PAI approach provides a prediction of an optimal treatment as well as an expected magnitude of differential outcomes across each treatment, thus providing clinicians with an actionable metric for clinical decision-making on treatment selection. The DeRubeis article applied the method to a cohort of patients with moderate to severe MDE enrolled in a comparative clinical trial comparing cognitive behavioral therapy (CBT) to paroxetine. Clinical and demographic data was used to predict individual PAI scores and identify an optimal treatment for each patient. Patients who were randomly assigned to their predicted optimal treatment had significantly more reduced symptoms compared to those who had been randomly assigned to their predicted non-optimal treatment. Subsequent studies have applied the PAI method to identify prescriptive predictors of CBT versus interpersonal psychotherapy¹⁷ and face-to-face CBT versus blended internet-based CBT¹⁸. To date, however, the PAI method has not been applied to predict treatment allocation for interventional psychiatry treatment options for sicker and more treatment-resistant patients who already failed several fist-line therapies, including electroconvulsive therapy (ECT) and (es)ketamine (I.V. ketamine and esketamine).

ECT is a rapidly acting and highly effective treatment for TRD. Previous studies have reported that patients with psychotic features of depression and older patients are generally most responsive to ECT¹⁹. Ketamine is an N-methyl-d-aspartate receptor antagonist commonly used for anesthetic purposes. Subanesthetic doses of ketamine administered at 0.5 mg per kilogram body weight have more recently been used as an effective and rapidly acting treatment for MDD and TRD^20,21. The S-enantiomer of ketamine (Esketamine) was approved in 2019 by the Food and Drug Administration for TRD in an intranasal formulation. Well-replicated predictors of antidepressant response to ketamine have included a higher body mass index and a positive family history of alcohol abuse^13,14. A recent open label, randomized, noninferiority trial compared antidepressant response rates between ECT (n = 170) and ketamine (n = 190; administered twice weekly for three weeks) in patients with TRD without psychosis and reported that ketamine was non-inferior to ECT²².

Interventional psychiatry (IP) is an emerging subspecialty that encompasses procedure-based treatments for patients with neuropsychiatric conditions²³. Treatments include device-based neuromodulation interventions such as ECT, but also advanced psychopharmacology treatments such as ketamine or esketamine, as these are procedural in nature, administered via intravenous infusion or intranasal spray, and require close medical and psychiatric monitoring. Until very recently, treatment selection in IP for patients with TRD was driven by practical limiting criteria. Access was a major factor: e.g., in certain areas only one treatment may be available. Another factor was patient preferences or knowledge: one may be frightened by the stigma of ECT or averse to the possible dissociative experience with (es)ketamine. Clinician preferences and education also played a role. In the last few years however, these obstacles have mostly disappeared; there is a growing number of clinics in academic and non-academic centers offering multiple IP treatments, staffed by clinicians trained across IP modalities, and with well-informed patients, or at least able to receive thoughtful recommendations. As the practical limiting obstacles that drove treatment selection decisions are becoming less relevant, we still do not have alternative evidence-based selection strategies, posing significant challenges for clinicians and patients: how should treaters decide the optimal strategy (ECT or (es)ketamine) for a given patient with TRD? This is particularly concerning given the common severity of TRD patients seeking IP treatment: suicide risk is high, and delaying effective treatment further burdens the patient (quality of life), their family (relationships), and the community (economic). Thus, determining patient-specific treatment selection algorithms is critical to support appropriate treatment recommendations leading to faster recovery.

In this study, we adapted and applied the PAI framework to a large, retrospective cohort of patients who underwent ECT or (es)ketamine to treat TRD to generate individual predictions of patient outcomes for each treatment using pretreatment medical records and demographic measures. Following earlier studies using the PAI method, we hypothesized that patients who received a treatment predicted to be optimal for them would have significantly lower depressive symptoms following treatment compared to those who received the sub-optimal treatment.

Results

Cohort characteristics

Demographic and clinical measures for the matched sample used for the main analysis are outlined in Table 1. Matching resulted in a sample of n = 392 patients (n = 196 per treatment arm). The mean baseline Quick Inventory of Depressive Symptomatology (QIDS)²⁴ scores were 17.76 ± 3.47 and 17.65 ± 3.48 for the ECT and (es)ketamine groups, respectively. The minimum QIDS score achieved by patients over the acute course of treatment (min-QIDS) scores was 11.33 ± 4.36 for the ECT and 10.64 ± 4.67 for the (es)ketamine cohorts. Neither baseline nor min-QIDS scores differed between cohorts. Average patient age was 42.78 ± 15.91 and 41.77 ± 16.07 years for the ECT and (es)ketamine groups, respectively, and did not differ significantly. Patient sex did not significantly differ between groups with 87 (44.4%) males in the ECT cohort and 75 (38.3%) males in the (es)ketamine cohort. The prevalence of bipolar disorder was significantly higher in the ECT cohort. Meanwhile, there was a higher number of medications taken prior to treatment in the (es)ketamine cohort. Baseline and min-QIDS scores did not differ between the IV ketamine and esketamine cohorts.

Table 1 Clinical and demographic characteristics of matched sample

Full size table

Matching diagnostics

Our primary matching strategy using baseline QIDS, age, psychotic symptom severity, and inpatient status resulted in well-balanced distributions of propensity scores between treatment groups and covariate standardized mean differences (SMD) well below 0.1 except for the Behavior and Symptom Identification Scale (BASIS)²⁵: psychosis score (SMD = 0.13). Associations between PSM variables with treatment group and min-QIDS scores are reported in Supplementary Results. Balance and Love plots for our primary matching strategy highlight differences between our matched and unmatched sample in Supplementary Fig. 1. Including prior (es)ketamine use to matching criteria resulted in unacceptable propensity score balance and SMDs. Prior (es)ketamine use was significantly associated with treatment type in the unmatched sample (χ² = 853.06, df=1, p < 0.001), however, prior (es)ketamine use was not associated with baseline or min-QIDS scores (see Supplementary Results). Dropping all patients with a prior history of (es)ketamine use and ECT patients who received ketamine during ECT prior to matching resulted in well-balanced propensity scores and SMDs below 0.1. Diagnostics of alternative matching strategies are outlined in Supplementary Results and Supplementary Figs. 1, 2. Matching to the IV ketamine-only cohort resulted in well-balanced propensity scores with all covariate SMDs below 0.1 (range = −0.08 to 0.08). Matching to the esketamine-only cohort resulted in well-balanced propensity scores between groups but large SMDs between covariates (range = 0 to 0.4) with only inpatient status being within an acceptable SMD range; see Supplementary Fig. 3.

Personalized Advantage Index

The global random forest regression (RFR) model predicted min-QIDS significantly above chance (mean R² = 0.18, p < 0.001). ECT was predicted to be optimal for 54 (27%) of patients who received it while (es)ketamine was predicted to be optimal for 137 (69%) of patients who received it. The average min-QIDS score for patients who received their optimal treatment (n = 191, 48%) was 10.62 ± 4.67 while the average min-QIDS score for those who received their non-optimal treatment was 11.32 ± 4.36. Across the whole sample, the difference in min-QIDS scores was not significantly different between patients who received an optimal versus non-optimal treatment (p > 0.05). At a PAI threshold of 0.3, resulting in a subset of 287 patients (72.4% of patients, n = 140 ECT and n = 147 (es)ketamine), patients who received their optimal treatment had a significantly lower min-QIDS score compared to those who received a non-optimal treatment (mean difference [MD] = 1.19 [95% CI: 0.32, ∞], t = 2.25, df=283.24 q < 0.05, Cohen’s d [d]=0.26). Higher PAI thresholds resulted in significant differences. For example, for the 18% of patients with a PAI of at least 1.5, a medium effect size difference in min-QIDS scores was observed, in favor of the optimal treatment group (MD = 2.49 [95% CI: 0.32, (infty)], t = 2.77, df = 68.02, q < 0.05, d = 0.53). Figure 1 illustrates mean differences in min-QIDS scores and effect sizes between patients who received optimal versus non-optimal treatments across a range of PAI thresholds for our main model and sensitivity analysis models. Power analysis revealed that the sample was powered to identify a small Cohen’s D effect size of d = 0.25.

Prognostic predictors

In descending order, the most important variables in the overall prediction of min-QIDS were pretreatment QIDS, treatment type, BASIS: self-harm score, BASIS: psychosis score, age, sex, BASIS: relationships score, BASIS: emotional lability score, diagnosis of a personality disorder, and the number of pre-existing diagnoses (neurological, psychiatric, general health). Figure 2(a) illustrates the overall feature importance scores for the top 10 most predictive features. SHAP (SHapley Additive exPlanations)²⁶ waterfall plots illustrate the contributions of individual features to the prediction of individual patient outcomes for a selection of two patients in Fig. 3, illustrating how this model might be used to inform clinical decision-making.

**Fig. 3: Contributions of predictors to individual predicted outcomes.**

Prescriptive predictors

Analysis of SHAP interaction plots indicated that min-QIDS predictions varied as a function of treatment type for several important predictor variables. Patients with baseline QIDS scores between 11 to 18 (moderate to severe depression) were expected to have marginally better outcomes with (es)ketamine over ECT while patients with QIDS scores above 18 (severe to very severe depression range) were expected to have more reduced symptoms with ECT. Similarly, lower BASIS: self-harm scores predicted preferential outcomes for (es)ketamine while higher scores predicted better outcomes for ECT. Higher BASIS: psychosis scores predicted marginally better outcomes with ECT. Patients between the ages of roughly 40–60 were anticipated to have favorable outcomes with (es)ketamine while patients younger than 40 or over 60 were predicted to have more reduced symptoms with ECT. A sex-specific interaction was observed in which male patients were predicted to have favorable outcomes with ECT while female patients had preferential outcomes with (es)ketamine. As sex-by-treatment interactions were unexpected, we conducted a series of pairwise t-tests to determine whether distributions of age might differ across sex-by-treatment groups. We observed trend-level, uncorrected, lower ages in the cohort of women receiving (es)ketamine compared to women receiving ECT (p < 0.1) and trend-level lower ages among males receiving ECT compared to women receiving ECT (p < 0.1), suggesting the potential for higher-order interactions driving sex-specific interactions. Lower BASIS: Relationship scores predicted better outcomes with (es)ketamine while higher scores predicted favorable outcomes with ECT. Lower BASIS: emotional lability scores predicted better outcomes with ECT while higher scores predicted improved outcomes with (es)ketamine. Diagnosis of a personality disorder indicated better outcomes with ECT. Last, patients with 3 or more comorbid diagnoses were expected to have better outcomes with ECT. Figure 2b illustrates SHAP interaction plots highlighting our model’s expectations for individual patient outcomes as a function of pretreatment predictors and treatment.

Alternative matching and exclusion criteria

Matching on alternative subsets of variables yielded differing ranges of PAI values. Matching on baseline QIDS and age resulted in significantly lower min-QIDS scores in patients who received an optimal treatment (MD = 1.97 [95% CI: 1.21, ∞], t = 4.26, df = 374.74, q < 0.001, d = 0.43) without PAI thresholding. Details of the cohort excluding all patients with a prior history of (es)ketamine use as well as ECT patients who used ketamine as an anesthetic for their current treatment course are outlined in Supplementary Table 1. Using this cohort, and matching on baseline QIDS and age, we observed a significant reduction in min-QIDS in the optimal treatment group relative to the non-optimal group (MD = 1.72 [95% CI: 0.71, ∞], t = 2.81, df = 231.56, q < 0.01, d = 0.36); see Supplementary Fig. 4(a). Prognostic and prescriptive predictors are outlined in Supplementary Fig. 5. Further results using alternative matching and exclusion criteria are detailed in the Supplementary Results section.

Subgroup findings

Propensity score matching was used to match the subset of IV ketamine patients to ECT patients after excluding all patients with prior (es)ketamine use and ECT patients who used ketamine as an anesthetic. This yielded a sample of n = 100 IV ketamine patients and n = 100 ECT patients; see Supplementary Table 2. Using our primary matching and alternative matching strategies, no significant differences in min-QIDS scores were observed between patients who received a predicted optimal or non-optimal treatment. Matching on pretreatment QIDS and age, we observed a significant reduction in min-QIDS in the optimal treatment group versus the non-optimal treatment group among the 48% of patients with the highest PAI scores (MD = 1.71 [95% CI: 0.03, ∞], t = 1.69, df = 93.92, p < 0.05, d = 0.33); however, this did not survive adjustment for multiple comparisons (see Supplementary Fig. 4b). Patterns of prescriptive predictors identified in our primary (es)ketamine model within the IV ketamine subgroup are illustrated in Supplementary Fig. 6. Matching esketamine and ECT patients yielded a small sample of n = 20 esketamine and n = 20 ECT patients (see Supplementary Table 3) and no significant between-group differences were observed. Power analyses revealed that the IV ketamine cohort was powered to detect a small effect size of d = 0.35 while the esketamine cohort was powered to detect a large effect size of d = 0.80.

Discussion

Prediction of optimal treatment allocation for individual patients is a central research aim in psychiatry. We developed a machine-learning adaptation of the Personalized Advantage Index approach originally developed by DeRubeis¹⁶ to predict optimal treatment allocation between ECT and (es)ketamine for individual patients using pretreatment measures of clinical records and demographic data. In this observational study, we matched patients on baseline depression severity, age, inpatient status, and severity of psychotic symptoms to more closely mimic what would be observed in a clinical trial. In the matched sample, treatment efficacy was equal which echoes a recent study confirming the noninferiority of ketamine to ECT as an antidepressant treatment for non-psychotic depression²². No significant differences in min-QIDS scores were observed between patients who received optimal versus non-optimal treatments when we compared all patients. However, this is somewhat expected as a proportion of patients were predicted to have only marginal differences in outcomes between treatments as reflected by a small PAI score. In clinical decision-making, treatment recommendations from this system would likely be made for patients with large differences in expected treatment outcomes. Conversely, treatment choices for patients with smaller PAI scores would likely be guided more by accessibility or personal preference¹⁶. Following this expectation, the 72% of patients with the largest PAI scores exhibited significant differences in min-QIDS scores with those assigned to an optimal treatment having average min-QIDS scores 1.9–2.4 points lower than those assigned to a non-optimal treatment, constituting small to medium effect size differences. Notably, however, a meaningful change threshold on the QIDS scale has been reported to be 3.5²⁷, which is larger than the differences detected in this study.

We evaluated results when alternative matching criteria were set as a sensitivity analysis. We noted that matching only on baseline QIDS scores resulted in no differences in outcomes, suggesting that adjusting for known predictors of outcomes in (es)ketamine and ECT was needed to yield meaningful predictions. When we adjusted for only baseline QIDS and age, differences in min-QIDS scores were detected across the entire sample and required no PAI thresholding. Adding prior (es)ketamine use as matching criteria led to imbalanced propensity scores and large between-group differences in matching covariates and was thus not considered. It is likely that the inability to adequately match patients when including prior (es)ketamine use reflects a high degree of bias between treatment groups as it was predominantly those receiving (es)ketamine who had also previously received (es)ketamine. Inclusion of variables related to exposure is common practice in PSM²⁸ to reduce bias; thus, including prior (es)ketamine use as a variable in our primary model might be warranted, however, a simulation study of variable selection strategies for PSM by Brookheart reported that inclusion of variables related to exposure but not to outcomes tends to increase variance of the estimated exposure effect without reducing bias²⁹ and we observed that prior (es)ketamine use was not related to either baseline QIDS or min-QIDS scores. We alternatively evaluated the effect of excluding all patients with a prior record of (es)ketamine treatment (within 60 days of current treatment) as these patients may have simply opted to continue or resume a treatment that improved their symptoms previously, thus potentially biasing our findings. We further excluded ECT patients who used ketamine as an anesthetic, though anesthetic levels of ketamine have not been demonstrated to interact with ECT response³⁰. Here, between-group differences were not observed in our primary matching strategy but were present when matching alternatively on baseline QIDS and age. Whether differences in results using this approach are due to the removal of bias contributed by patients continuing (es)ketamine treatment and/or a reduction of statistical power resulting from n = 148 fewer patients in this approach, remains to be determined.

To our knowledge, this is the first application of the PAI method to predict treatment allocation outside of CBT, psychotherapy, or selective serotonin reuptake inhibitors (SSRIs)^17,18,31,32. An alternative strategy to predict individual likelihoods of treatment response to SSRIs, SNRI, bupropion, and mirtazapine treatments using similar electronic health record data was recently developed by Sheu et al.³³.

There is an urgent need to optimize antidepressant treatment selection, particularly for patients suffering from TRD, which is associated with extended and costly inpatient care^34,35. (Es)ketamine and ECT are two rapidly acting treatments for TRD with comparable efficacy in non-psychotic depression²². Patients may have personal preferences in selecting ECT or (es)ketamine which may be informed by factors including, for instance, that ECT requires general anesthesia and has been linked with transient memory impairment³⁶. (Es)ketamine, however, has liability for abuse^22,37 and is commonly not indicated for patients with psychotic features of depression. For patients with a negligible predicted difference in outcomes between these treatments, treatment selection may be informed by weighing these factors. Patients with a large differential in predicted outcomes, however, may also factor into their decision the expected difference in outcomes using this method.

Several factors informed preferential outcomes between ECT and (es)ketamine in this observational sample; we discuss several of these below. Our models predicted better outcomes for patients with moderate to severe depression who used (es)ketamine. Patients with severe to very severe depression, however, were expected to recover more with ECT. Pretreatment symptom severity is a commonly reported predictor of subsequent symptom change for both ECT and (es)ketamine with more severe symptoms generally predicting poorer outcomes^8,38,39. Relatedly, more severe self-harm was an indicator of preferential outcomes with ECT. Previous studies have reported that ECT reduces suicidal ideation^40,41 and attempts⁴², however, little has been reported about self-harm as a predictor of ECT response. An earlier report found that a prior history of suicide attempts predicted poorer outcomes for ketamine¹⁴, in line with our current findings. The BASIS scale captures several symptoms of psychosis including hallucinations and delusions. Our model predicted that patients with more severe psychotic symptoms would have better outcomes with ECT versus (es)ketamine. Patients with psychotic features of depression are widely reported to respond well to ECT^43,44 while psychotic symptoms are often exclusionary for treatment with (es)ketamine²². Notably, a full diagnosis of psychosis was exclusionary for (es)ketamine patients included in this study, however, some (es)ketamine patients did endorse symptoms of psychosis captured by the BASIS scale which was not exclusionary.

Older age has been widely reported as a predictor of improved odds of response or remission following ECT^19,44. Echoing this, our model predicted favorable outcomes for patients over roughly 60 years of age receiving ECT relative to (es)ketamine. Somewhat unexpectedly, younger patients, approximately ages 30 and below, were also predicted to have improved outcomes with ECT. Meanwhile, patients between the ages of roughly 40–60 years of age were predicted to have better outcomes with (es)ketamine compared to ECT. Our model predicted sex-specific improvements between treatments with women predicted to have better outcomes with (es)ketamine while male patients were predicted to have better outcomes with ECT. To our knowledge, however, previous studies have not identified sex-specific differences in remission/response rates following either ECT⁴⁵ or ketamine⁴⁶. Treatment-related symptom changes did not vary significantly by sex between treatments. One potential reason for this apparent discrepancy could be that higher-order interactions between treatment and sex are captured in our multivariate model. We observed, for example, that the average age was somewhat lower in women who received (es)ketamine compared to those who received ECT and somewhat lower in men compared to women who received ECT, which may suggest a sex-by-age-by-treatment expectation of our model.

Subjective perception of the quality of patient’s social relationships captured by the BASIS: relationships score was predictive of differential treatment outcomes. Patients with lower BASIS: relationship scores (indicating higher perceived quality of relationships) were predicted to have better outcomes with (es)ketamine. In general, social support has been reported to be associated with positive treatment outcomes in mood disorders⁴⁷. As it pertains to (es)ketamine, several reports have noted that subjective factors including the patient’s trust in and supportive relationships with clinical staff are important for ketamine-assisted psychotherapy^48,49,50. Higher emotional lability was predictive of improved outcomes with (es)ketamine compared to ECT. Existing literature relating emotional lability/reactivity to (es)ketamine outcomes is sparse, however, a recent neuroimaging study reported that increased activity of the pregenual anterior cingulate cortex during an emotional stimulation task was related to the antidepressant efficacy of IV ketamine⁵¹.

Our model also predicted improved efficacy of ECT for patients with a comorbid personality disorder. Previous studies have found that ECT is equally effective for depressed patients with comorbid borderline personality disorder (BPD)^52,53. Evidence is more limited for ketamine, however, a recent pilot study reported that ketamine reduced symptoms of BPD significantly compared to midazolam⁵⁴. The cumulative number of prior patient general health, neurological, or psychiatric comorbid diagnoses was differentially predictive of ECT and (es)ketamine outcomes. Generally, patients with a larger number of prior diagnoses were predicted to have better outcomes with ECT compared to (es)ketamine. A recent study reported that antidepressant response to IV ketamine is not impaired by psychiatric comorbidities including GAD, OCD, PTSD, and ADHD⁵⁵, although, safety and tolerability concerns are noted for patients with cardiovascular or metabolic comorbidities⁵⁶.

Because our primary analysis included ketamine patients who received either IV ketamine or intranasal esketamine, we conducted subgroup analyses matching ECT patients to ketamine and esketamine cohorts, separately. Differences in min-QIDS scores were seen between patients with a large predicted advantage for an optimal versus non-optimal treatment in the IV ketamine subgroup when patients were matched on baseline QIDS and age; however, these results did not survive adjustment for multiple comparisons. We also did not observe differences in min-QIDS in the esketamine subgroup. Notably, neither baseline nor min-QIDS scores differed between patients who received ketamine or esketamine. The absence of robust differences in our subgroup analyses may be a function of reduced statistical power with fewer observations and diminishing power at higher PAI thresholds which further reduce the number of observations used in each t-test. For the esketamine subgroup analysis, we also note that matching was inadequate at balancing between-group SMDs between covariates used for matching.

Preclinical research suggests that (S)- and (R)-ketamine operates on differing mechanisms⁵⁷, however, there is little literature comparing the antidepressant effects of racemic IV ketamine to esketamine. A recent study comparing IV ketamine to intranasal esketamine reported similar efficacy for both treatments⁵⁸, echoing observations in our cohort. In our subgroup analyses, there was a mixture of overlapping and unique predictors important in predicting min-QIDS scores. It is challenging to interpret differing patterns of prescriptive measures between our primary analysis and the IV ketamine subgroup analysis which yielded uncorrected significant between-group differences using a secondary matching strategy. However, in the IV ketamine subgroup sample matched on pretreatment QIDS and age, the most important predictors from our primary analysis followed similar prescriptive patterns including the BASIS: self-harm, emotional lability, psychosis, and relationship scores, as well as age and number of prior diagnoses. Contrary to our primary analysis, moderate depression was better indicated for ECT while severe depression was predicted to have better outcomes with IV ketamine, diagnosis of a personality disorder was better indicated for IV ketamine, and patient sex was not informative. These differences, however, must be weighed against the observation that subgroup analyses were less powered. Thus, additional studies are needed to resolve these potential differences in prescriptive markers.

It is notable that our approach differs from previous implementations of the PAI method. As noted by Huibers et al.¹⁷, earlier uses of the PAI method performed feature selection outside of cross-validation, on the whole data set, which is likely to introduce information leakage bias⁵⁹ and inflate model performance. Here, we predicted patient outcomes using random forest regression models which are a class of embedded methods, meaning that feature selection is integrated into the model’s training^60,61. We used nested cross-validation to tune model parameters and identify optimal subsets of predictors to avoid information leakage.

There are several important limitations to consider when interpreting these findings. Most PAI studies have had the advantage of using randomized clinical trial data which effectively minimizes the potential for introducing confounder bias. This study has used observational data and subsequent PSM to account for known confounds; however, PSM is unable to account for unknown/unmeasured confounds that may influence either naturalistic treatment allocation or subsequent outcomes^62,63. To partially address this limitation, we matched patients on expected confounds including pretreatment symptom severity, age, inpatient status, and severity of psychotic symptoms. Using sensitivity analyses, we explored the results of our models if matching criteria were altered and observed a mixed pattern of significant and trend-level differences in min-QIDS scores between patients who received a treatment predicted optimal had significantly better outcomes. However, this was not true when matching only on baseline symptoms. Our observational sample also included a proportion of (es)ketamine patients with a record of using (es)ketamine within 60 days before their current treatment course which may bias results. We excluded these patients and ECT patients who received anesthetic doses of ketamine in a sensitivity analysis which eliminated the observed between-group differences in our primary approach that included these patients. This may be due to a joint reduction of statistical power and bias removal, but additional studies with a larger sample of (es)ketamine patients will be required to resolve this. Notably, however, matching on pretreatment QIDS and age resulted in significant between-group differences despite patient exclusion. (Es)ketamine patients in our primary analysis were a mixture of IV ketamine and esketamine with a minority of patients missing clear records for IV ketamine versus esketamine. However, pre- and post-treatment symptom severity did not differ between (es)ketamine subgroups. Between-group differences were not observed in the IV ketamine subgroup analysis using our primary matching strategy but were present when alternatively matching on baseline QIDS and age. Here, barring a few exceptions, prescriptive predictors were generally similar in directionality in this subgroup analysis. Differences between the primary analysis and subgroup analysis may reflect reductions in overall statistical power. Furthermore, propensity score matching was not reliable in the smaller esketamine cohort. Relatedly, the proportion of inpatients was higher in the (es)ketamine group compared to ECT in our sample prior to matching. In broader clinical practice beyond McLean Hospital, (es)ketamine is predominantly an outpatient procedure, and this difference may affect model generalizability. Another notable consideration is that the expected differences in min-QIDS scores between optimal and non-optimal treatment cohorts were 1.9 to 2.4, on average. A clinically significant change on the QIDS-SR scale, however, has been reported to be 3.5²⁷. Despite this, our findings support that allocation of a predicted optimal treatment results in significant symptom reductions ranging from small to medium effect sizes. We additionally risked overcorrection of tests for between-group differences in min-QIDS scores. Specifically, multiple comparison corrections were applied across a range of PAI scores at arbitrary intervals using increasingly smaller subsets of the same patient cohort. As a result, comparisons are non-independent and adjusted stringently. Additional limitations derive from the retrospective observational nature of the study. For instance, study inclusion required the patient to be able to complete baseline severity measures, which may exclude the most severely ill patients who were unable to complete these assessments. Furthermore, the diagnosis was determined by the treating clinician rather than by structured interview, which may hinder comparisons with prospective trials but matches ordinary clinical practice. Additionally, we cannot control for the effects of concomitant medication changes or psychotherapy treatment that may have occurred during the study period.

Predicting which antidepressant treatment will elicit the most robust response from an individual patient is of the utmost importance. In this study, we adapted the PAI method to predict optimal treatment allocation between two equally effective rapidly acting treatments for TRD: ECT and (es)ketamine. As hypothesized, patients who received a treatment predicted optimal had significantly better treatment outcomes reflecting small to medium effect size differences. Importantly, these models were constructed using commonly acquired and inexpensive demographic and medical record data. Future work will expand this approach to additional treatments including transcranial magnetic stimulation, selection of optimal treatment parameters, and evaluate the generalizability of this model to patients from external clinics. We observed that matching criteria strongly affected model performance. A systematic evaluation of how matching strategies effect model performance and generalizability to new data will be critical to translate models like this to clinical practices. Precision medicine methods such as this have the potential to provide actionable predictions for both patients and clinicians in the selection of treatments and their use should be expanded to include additional treatment modalities.

Methods

Participants

Medical record data on 2671 patients who underwent ECT (n = 2526) or (es)ketamine (n = 235) at McLean Hospital between 2011 and 2022 was aggregated from EPIC and mapped to medical records at McLean. Ketamine patients received either IV ketamine (n = 190) or intranasal (IN) esketamine (n = 43) with the remaining 2 (es)ketamine patients having unidentifiable IV or esketamine treatments. Patients were included in this study if they had a clinical diagnosis of depression (unipolar or bipolar). Diagnosis was made by the Referring psychiatrist or Psychiatric nurse practitioner and confirmed by the Consulting MD during clinical assessment. Patients were treated with either ECT or (es)ketamine at the study site. Fifty-nine patient records were from second or third treatment courses (n = 20 ECT and n = 39 (es)ketamine) in which patients switched from ECT to (es)ketamine or vice versa and were therefore removed such that only patient records from their first recorded treatment course at McLean was used resulting in a sample of n = 2506 ECT patients and n = 196 (es)ketamine patients. No patients under 18 years of age were included. A diagnosis of psychosis was exclusionary for the (es)ketamine cohort, however, a minority of (es)ketamine patients endorsed symptoms of psychosis captured by the BASIS psychosis⁶⁴. Depressive symptoms were assessed using the QIDS Self Report Scale⁶⁵. Patients were assessed over the acute phase of treatment, defined as the first 10 treatments for ECT and the first 8 treatments for (es)ketamine. ECT patients completed the QIDS before the first treatment and after every 5^th subsequent treatment. (Es)ketamine patients completed the QIDS prior to each treatment. Patients were included in the cohort if the pre-treatment QIDS ≥ 10, indicating at least moderate depression severity. Treatment was provided as part of routine clinical care, as described below. This retrospective analysis of clinical records was approved by the Mass General Brigham IRB with a waiver of informed consent.

Participant matching

Earlier studies using the PAI method have used data on randomized controlled trials to avoid potential confounds arising from naturalistic settings in which patient characteristics inform treatment selection. Data used in this study is based on naturalistic patient assignment to either ECT or (es)ketamine and, as such, patient symptom severity, proportion of inpatients, psychotic features of depression, and age, differed significantly between arms. To adjust for this, we used propensity score matching (PSM)²⁸ to match patients between treatments on pretreatment QIDS scores, inpatient status, severity of psychotic features captured by the 24-item BASIS psychosis subscale⁶⁴, and age. Variables used for PSM were required to be associated with both treatment and the outcome variable (min-QIDS)²⁹. Matching was done using a 1:1 ratio of patients across treatments without replacement to prevent duplicates using nearest neighbor matching computed from propensity scores using the MatchIt library⁶⁶. Calipers were not used to constrain matches. Propensity score balance was inspected using balance plots before and after matching while standardized mean differences (SMD) of matching covariates between treatment groups were assessed using Love plots using the Cobalt library⁶⁷. While no consensus on thresholds for SMDs of covariates between groups exists, common guidelines suggest that SMDs of 0.1 or greater indicates potential imbalance between samples⁶⁸. Matching yielded a final sample of 392 patients (n = 196 per treatment). An outline of patient characteristics after matching is provided in Table 1. A summary of patient characteristics prior to matching is provided in Supplementary Table 4.

Electroconvulsive therapy

ECT was provided using a Mecta Spectrum 5000Q instrument (Tualatin, OR) with individualized seizure threshold determination at the time of first treatment, as previously reported⁶⁹. Subsequent treatments were delivered initially at 6× seizure threshold for right unilateral treatments, typically three times weekly. Electrode placement and ECT charge were then adjusted based on response by the treating psychiatrist⁷⁰, with details of the clinical treatment course previously reported. Methohexital was the default anesthetic, although etomidate, propofol, or ketamine anesthetic could be used at the discretion of the treating anesthesiologist. Muscle relaxation was provided by succinylcholine.

Ketamine and esketamine treatment

IV ketamine and esketamine treatment were offered to patients with TRD defined by a history of two or more treatment failures with standard anti-depressants at adequate dosing and duration (as best could be determined in a naturalistic, clinical setting and in collaboration with their outpatient referring provider). Patients with a history of psychosis, current substance use disorder, and relevant uncontrolled medical (i.e., arteriovenous malformation, uncontrolled HTN, aneurysmal disease) were deemed ineligible. Prior to initiation of treatment, each patient is evaluated by an affiliated Internal Medicine physician or NP to obtain medical clearance. At each treatment, patients were evaluated and monitored by a ketamine-trained staff psychiatrist, and a ketamine-trained nurse. Patients were monitored with pulse oximetry, automated Blood Pressure monitoring and one:one nursing care in private, low-stimulation treatment rooms. An Anesthesia physician was available if needed.

Patients treated with I.V racemic ketamine began at a standard initial dose of 0.5 mg/kg administered over 40 min. Patients who failed do demonstrate clinically meaningful response by integrated subjective report, objective clinical assessment, and evaluation of QIDS scores were advised to cease further treatment following treatment #3-#4. The dose was adjusted at the staff psychiatrist’s clinical discretion (but could not exceed 1.0 mg/kg) over the course of the treatment series according to clinical response. A full course of IV racemic ketamine was defined as 8 total treatments.

Esketamine was delivered in accordance with the 3-phase protocol with REMS monitoring mandated by the product label. All participants received training and practiced using the intranasal device before the first administration. Participants self-administered intranasal study drug at the clinical site under the direct supervision of the esketamine-trained nurse. Most participants received the first dose of 56 mg with the possibility of increasing the dose to 84 mg contingent on patient tolerability to the index dose and according to the patient’s response. All participants were monitored at the clinic for to 2 h following treatment.

During the treatments, other pharmacological and psychotherapeutic treatments were continued as part of the usual regimen. During and after the procedure, patients who experienced nausea could receive ondansetron. Metoprolol or Versed was available for blood pressure control. Criteria for discharge readiness included a return to baseline mental status, absence of gait disturbance and nausea, and normal blood pressure. Any administration required the patient to be discharged to the care of an adult escort.

Clinical and demographic predictors

Predictors for our models were 112 demographic and pretreatment medical record measures including a treatment label ((es)ketamine or ECT), the 24-item BASIS relationships, self-harm, emotional lability, psychosis, and substance abuse subscales⁶⁴, the Montreal Cognitive Assessment (MoCA) scale⁷¹, indicator variables for medication history, and comorbid neurological, psychiatric, or general health diagnoses, self-reported race (White, Black or African American, Asian, or other) and ethnicity (Hispanic or Latino), age, and self-reported sex. A tabulation of predictor variables is given in Supplementary Table 5. Importantly, these are measures commonly used in clinical practice, easy to capture, and scalable at the population level.

MoCA and BASIS subscale scores were missing for a subset of patients and filled using imputation within the treatment group. BASIS subscales were missing for roughly 2–3% of the ECT or (es)ketamine group while 29% of MoCA scores were missing for the (es)ketamine cohort. Missing MoCA scores and missing BASIS subscales were imputed within treatment classes. Missing MoCA scores, BASIS: psychosis and substance abuse scores were filled using the mode due to distributional skews. BASIS: emotional lability, relationship, and self-harm scores were imputed using mean due to the more normal or uniform distributions.

Calculation of the Personalized Advantage Index

To calculate PAI scores, we trained a series of random forest regression (RFR) models to predict min-QIDS, which was drawn from the 5th or 10th treatment for ECT patients and 2nd to 8th treatment for (es)ketamine patients. All 112 pretreatment clinical and demographic variables described above were included as predictors. Predictions of min-QIDS scores were made using leave-one-out cross-validation (LOOCV) wherein models were trained on n−1 participants and the fitted model was used to predict min-QIDS in the nth held-out participant. Each RFR model had 1000 underlying regression trees and was fit using 10-fold cross-validation with nested feature selection and a grid search which were embedded sequentially in a secondary tenfold cross-validation applied to the training data. Feature selection proceeded in two steps: first, a near-zero-variance filter was used to remove variables in the training data that had only one unique value or a high ratio (95:5) of the most frequent value to the second most frequent variable value. Second, the remaining features were ordered based on their permutation-based importance scores⁷² derived from a nested RFR model trained to predict min-QIDS in the training data. The number of features retained was the minimum of the number of features with non-zero importance scores or the upper 70% of the most important features. RFR parameters optimized in the nested grid search included mtry (the number of variables to consider split for each node split), splitrule (a function to evaluate the quality of each potential node split), min.node.size (the minimum number of samples in a node to allow a further split), and n.filter (the number of features to retain following feature selection). Random forest models were fit using the ranger package⁷³ and nested cross validation was implemented using the nestedcv package⁷⁴ in R version 4.3.0⁷⁵.

As outlined in the DeRubeis article¹⁶, two min-QIDS predictions were made for each held-out patient: one using the patient’s true treatment label and a counterfactual prediction in which the treatment label was switched to the treatment the patient did not receive. The prediction resulting in the lowest min-QIDS score was deemed the patient’s predicted optimal treatment while the prediction resulting in the larger min-QIDS score was deemed the patient’s non-optimal treatment. The magnitude of the difference between the optimal and non-optimal min-QIDS predictions is referred to as the patients predicted advantage: the PAI score. To test the hypothesis that patients who received their predicted optimal treatment would have lower min-QIDS scores following treatment, we compared distributions of min-QIDS scores between patients who received their predicted optimal treatments to those who did not using a two-sample, one-sided Welch t-test. Because our hypothesis is directional, a one-sided t-test is justified.

In clinical practice, however, a patient with a PAI score near zero would not be expected to respond preferentially to one treatment over another; thus, treatment selection would be determined by other factors such as personal preference. Therefore, we examined differences in outcomes between patients who received optimal versus non-optimal treatments in the subset of patients with increasingly higher PAI values, from 0 to the maximum PAI score in steps of 0.1 stopping when fewer than 30 patients remained in either the optimal or non-optimal treatment groups after thresholding to limit comparisons made with smaller samples. This constraint was reduced to 20 patients per group and removed for exploratory subgroup analyses of IV ketamine and esketamine cohorts, respectively, due to smaller sample sizes. T-tests across this range of PAI scores were conservatively adjusted for multiple comparisons using an FDR adjustment though tests were non-independent given the overlap of patients across PAI thresholds.

Model evaluation

A global RFR model was fit to the whole dataset to interpret the contributions of features and feature interactions in the prediction of min-QIDS. This model was trained using the same steps described above without the outer LOOCV loop. The performance of the global RFR model and the series of RFR models generating PAI scores were evaluated using the sum of squares formulation of the R² (coefficient of determination) measure⁵⁹ which describes the fraction of explained variance in the min-QIDS measure captured by our models. The significance of the R² scores was assessed using permutation tests with B = 1000 permutations of the entire modeling procedure in which the min-QIDS score was randomly reshuffled across patients at each iteration.

Evaluation of prognostic and prescriptive measures

Earlier studies applying the PAI method have distinguished between prognostic and prescriptive predictors. Here, prognostic predictors refer to baseline predictors that are indicative of an individual patient’s outcome following treatment or a set of treatments but do not indicate which treatment is expected to yield an optimal outcome. Prescriptive variables, in contrast, predict outcomes as a function of treatment type and can therefore inform optimal treatment selection¹⁷.

The contributions of predictive features to model predictions at the group and individual patient level were evaluated using SHAP analysis²⁶ in the R-based treeshap package⁷⁶. Through inspection of SHAP plots, we investigate three properties of our global RFR model: 1) prognostic predictors through evaluation of the overall importance of each feature in the prediction of min-QIDS and the directionality of important features with respect to predicted outcomes; 2) prescriptive predictors through inspection of SHAP interaction plots illustrating expected changes in outcomes that vary as a function of a predictor’s value and treatment type (ECT versus (es)ketamine); and 3) decision paths for individual patients illustrating how observed values of their pretreatment clinical and demographic characteristics produced their predicted treatment outcome.

It is notable that earlier PAI implementations have investigated prescriptive predictors by including interaction terms between pretreatment predictors and treatment type. Tree-based regression/classification models such as RFRs, however, detect interaction effects through optimization of decision tree paths in which the influence of a given variable is conditioned on the value of preceding variables in the decision tree^60,77. Thus, we did not directly include interaction terms as model predictors but recovered them from the global RFR model through analysis of SHAP interaction values⁷⁶.

Sensitivity analyses

Propensity score matching (PSM) is a useful method for adjusting for known confounds in observational studies; however, it is not a perfect substitute for randomization as it cannot adjust for unmeasured confounds^62,63. Our main analysis used PSM to match patients on pretreatment QIDS scores, psychotic symptoms from the BASIS-24 psychosis subscale, inpatient status, and age. This approach was taken because psychotic features of depression have been identified as predictive of response to ECT⁴⁴ and are often exclusionary for (es)ketamine treatment⁷⁸. The prevalence of psychotic features of depression differed significantly across treatment arms after matching baseline QIDS. Similarly, the proportion of inpatients was higher in the (es)ketamine group. Moreover, each of these baseline measures was significantly related to min-QIDS scores. To evaluate the sensitivity of our model’s performance to this choice of matching criteria, we repeated our analysis using several subsets of our matching criteria: 1) matching only on baseline QIDS; 2) matching on baseline QIDS and patient age; and 3) matching on baseline QIDS, age, and inpatient status. Additionally, because the proportion of prior (es)ketamine differed between groups and prior (es)ketamine use among these patients may indicate continued use of a treatment known to work well for the given patient, we added an additional matching criterion of prior ketamine or esketamine use to our primary matching strategy. Lastly, we evaluated our models by first dropping patients with a prior use of (es)ketamine and ECT patients who received ketamine as an anesthetic in their current ECT course. Notably, anesthetic doses of ketamine have not been shown to effect the antidepressant effects of ECT³⁰, so we also repeated this sensitivity analysis while including ECT patients who received ketamine as an anesthetic. The relationship of PSM covariates with min-QIDS was evaluated using the unmatched sample using two-sided t-tests for categorical variables and Pearson’s correlation for continuous variables.

Subgroup analyses

Ketamine patients were a mixture of IV ketamine and intranasal (IN) esketamine. We repeated the above analyses using subsets of patients who received IV ketamine or IN esketamine. ECT patients were matched to each ketamine group using the same PSM matching procedures described previously and all patients with a prior record of (es)ketamine use as well as ECT patients receiving anesthetic ketamine were excluded to mitigate potential bias.

Power analyses

We conducted power analyses for t-tests to identify the minimum effect size detectable given each sample size for the primary and subgroup analyses. Power was reported for two-sample, one-sided t-tests with a significance level of 0.05 with 80% power to detect a significant effect.

Comparison of treatment arms

Patient characteristics were compared between treatment arms using t-tests and Chi-squared tests where appropriate.