Association of Use of Electronic Appointment Reminders With Waiting Times in the Veterans Affairs Health System

Key Points Question Is use of an electronic appointment reminder system associated with the number of days patients have to wait to complete their appointments? Findings In this cohort study of 5 116 085 patients and 102 249 484 bookings in the Veterans Affairs Health System, the introduction of an electronic appointment reminder system was associated with a mean reduction in waiting time of 6.51 days compared with a baseline wait of approximately 60 days. Meaning Results of this study suggest that use of an electronic appointment reminder system may be associated with improved patient access to health services.


Ordinary Linear Regression Models
We begin by describing Models 1 and 2 of our primary analysis of this paper, which are ordinary multivariable linear regressions. Mathematically, they are represented as follows: Model 1: = 0 + + ′ + Model 2: = 0 + + + × ( × ) + ′ + In these expressions, scalar quantities are denoted in regular typeface, whereas vector quantities are denoted in boldface. The prime (') symbol represents vector transposition.
In both models, the symbol is used to index all completed appointments, and the dependent variable, , refers to the waiting time of the th completed appointment, which is defined as the number of days between the date that the appointment is booked (booking date) and the date that the appointment is completed (appointment date).
The error term captures random noise that we assumed to be clustered at the clinic level, i.e., the level of the intervention variable. In other words, within all observations for a given clinic, arbitrary correlations were allowed in these terms. This noise model only assumed that these error terms were uncorrelated across different clinics. Allowing for this correlation typically leads to more conservative estimates (i.e., larger standard errors) than assuming that errors are independent across all observations.
The key independent variable in both models is the binary-valued intervention indicator, . This takes the value = 0 if the appointment date of the th completed appointment comes before the clinic (at which the appointment was made) adopted VEText. Conversely, this takes the value = 1 if the appointment date of the th completed appointment falls after the clinic had adopted VEText and the patient had received at least one VEText reminder.
Model 2 includes another independent variable , which refers to the number of incomplete bookings (i.e., were not completed due to cancellation or no-shows) associated with the completed appointment . To be included in this count, the incomplete booking had to be made by the same patient for that same appointment class, and had to have a booking date that was between the booking date and appointment date of the completed appointment .
Model 2 also contains an independent variable that is the interaction between the intervention indicator and the number of incomplete bookings, × . We include this interaction to assess the change in association between the number of incomplete bookings and waiting time that is, in turn, associated with the VEText intervention. This comes from the estimated regression coefficient × because we can group terms as follows + × ( × ) = ( + × ) , from which this interpretation of the coefficient × should be more apparent.
In both models, the vector represents the vector of controls associated with the th completed appointment, which can be categorized into different sets of controls, listed in eTable 1. Full results for these regressions are reported in the first two columns of eTable 2. The IV regression model specifications for Models 3 and 4 parallel those for Models 1 and 2 respectively. The key difference between the IV regressions and ordinary linear regressions is that each IV regression model involves estimating parameters for two sets of equations, which are respectively called the "selection equation(s)" and the "outcome equation". Model 3 has one selection equation and one outcome equation. Model 4 also has one outcome equation, but because this outcome equation includes an interaction term that involves the intervention indicator, we require two selection equations. 2, Chap 4.6.1 These are described below for each model. As with the ordinary linear regressions, the symbol is used to index all completed appointments, and all other previously defined symbols retain their original definitions. We proceed to describe the new symbols introduced in these models.
The instrumental variable in both these models is represented by the binary-valued . This was assigned the value = 1 if the appointment date of the completed appointment was scheduled on the third week or later of the centrally determined rollout date for that clinic, and = 0 otherwise. This was because each wave of the rollout was designated to have two full weeks of implementation.
The error term captures random noise in the first (and, in the case of Model 3, only) selection equation of each model. For Model 4, the error term captures random noise in the second selection equation. Like the terms , both these error terms were assumed to be clustered at the clinic level.
From these models, the classical procedure for instrumental variable regression occurs in two steps. 2, Chap 4.6.1 We first describe the procedure for Model 4. First, estimates of the coefficients of the selection equations (̂0,̂,̂× ,̂) and (̂0,̂,̂× ,̂) are generated using ordinary linear regression. Next, using these coefficients, and for each observation, the estimated intervention ̂ and the estimated intervention interaction term × are constructed using these estimated coefficients as follows: ̂= ̂0 + ̂+ ̂× ( × ) +̂′ × = ̂0 + ̂+ ̂× ( × ) +̂′ In the second step, the dependent variable is regressed now using the estimated intervention variable and interaction term ̂ and × in place of the actual intervention variable and interaction term and × , with the other independent and control variables left unchanged. Through this process, the estimated coefficients from the second estimating equations of both models would then be properly adjusted for self-selection bias.
The procedure for Model 3 is analogous, except that there is only one selection equation. In the first step, only (̂0,̂,̂) is estimated. This is then used to construct the estimated intervention variable ̂ in the second step.
Results for these regressions are reported in the rightmost two columns of eTable 2. In addition, eTable 3 presents the main results for these regressions as groups of control variables are progressively added into the model. Finally, for Model 4, which is the fully controlled model with incomplete bookings and interactions added as regressors, we report first-stage regression statistics as well as second-stage regression results in eTable 4, both in aggregate (first column) and by appointment groups (second to fifth column).
The first stage Kleibergen-Paap rk Lagrange Multiplier (LM) statistics indicate that the null hypothesis of underidentification should be rejected at the conventional 5% level, which provides evidence that the instrumental variable is indeed correlated with the treatment variable . 3 Another validity threat to an IV regression can occur if this correlation between and is non-zero, but weak (i.e., small in magnitude). Two statistical tests suggest that this is not the case in our setting: (a) The Kleibergen-Paap rk Wald F-statistics are substantially larger than the conventional rejection threshold of 10; and (b) the Cragg-Donald Wald F-statistics are also much larger than the Stock-Yogo critical values that conservatively assume a bias of 10%. 3,4 Finally, with the exception of the Rehab group, the C-statistics indicate that the null hypothesis of treatment exogeneity should be rejected at the conventional 5% level. 3 Intuitively, these results mean that it would be inappropriate to perform ordinary linear regression by simply including the treatment as a regressor. This further substantiates our use of a two-stage IV regression.

Regression Models for Post-hoc Analyses
In post-hoc analyses, we sought to investigate whether the introduction of VEText was associated with changes in the average number of cancellations per completed appointment. As described in the main text, we conducted these analyses to assess whether VEText might be plausibly associated with a more tightly packed appointment schedule, which would be likely if VEText was found to be associated with fewer cancellations.
These analyses were also conducted as IV regressions and follow the same two-step procedure described above. Their specifications parallel that of Model 3 in our primary analysis, and can be written as follows: The unit of observation remains the same as in Model 3. The key difference stems from the dependent variable and this is reflected by our choice of a different symbol. In our main analysis, the dependent variable was waiting time, which we had denoted by . In this post-hoc analysis, the dependent variable was the number of patient-cancelled bookings, which we denote by . We conducted this regression analysis on the full sample, and repeated this analysis, stratified by clinical group. As described in the main text, we also conducted additional analyses where we separately investigated modifications of the dependent variable to the number of patient-cancelled bookings within 7, 14, and 21 days respectively.
Full results of these analyses are reported in eTables 7-10. These results reveal that the introduction of VEText was associated with a decrease in the average number of cancelled bookings, and average number of bookings that were cancelled at short notice, within 7, 14, and 21 days. These directional associations were found for all clinical groups in aggregate as well as for each clinical group.

Sensitivity and Subgroup Analyses
We conducted a suite of sensitivity and subgroup analyses to assess the robustness of the results to different model specifications and across different strata of the data. These were all based on IV regression Model 4, both across all clinical groups in aggregate and further stratified by clinical groups. We describe each of these analyses below.

Sensitivity Analyses
Sensitivity Analysis 1: In this group of analyses, we explored whether our results were robust when the maximum allowable waiting time was cut off at 12 months, 9 months, 6 months, and 3 months, respectively. We performed this analysis because it was possible that some appointments that were made far in advance could be routine appointments for which changes in waiting time would be less clinically meaningful.

Sensitivity Analysis 2:
In the main regression analyses of our paper, the calendar month of the appointment was coded as a categorical variable. In this sensitivity analysis, we investigated how sensitive our results were to this specification, by changing month to a continuous variable.

Sensitivity Analysis 3:
The IV approach used in our main regression analyses is technically called the standard two-stage least squares (2SLS) estimator, because it models both selection and outcome stages as linear regression problems. Although this approach has several appealing statistical properties, it is somewhat inefficient for our present problem because it doesn't exploit the fact that the treatment is binary-valued.
A known way to incorporate this information is by a simple modification of this setup that uses nonlinearly-fitted treatment probabilities to replace our original instrumental variables. 2, Chapter 4.6.1 To do this, we first construct a nonlinear regression model of the treatment on the instrument that exploits the binary nature of the data (such as a logistic or probit regression). For concreteness, we use the probit regression model specified below: Where Φ(⋅) represents the cumulative distribution function of the standard normal distribution. Maximum likelihood estimation is then used to estimate the coefficients of this regression (̂0,̂,̂× ,̂), from which we can obtain the fitted probabilities of assignment to treatment: Having obtained these fitted probabilities, we then implement the same 2SLS in our original approach, with the exception we use these fitted probabilities ̂ in place of our original instrument .
The results of these analyses are summarized in eTable 5, which shows that the main results were mostly directionally robust to these alternative modeling assumptions and structures.

Subgroup Analyses
Subgroup Analysis 1: We conducted age-stratified analysis by dividing the sample into two age groups (65 and below vs 66 and above).

Subgroup Analysis 2:
We conducted race-stratified analysis by dividing the sample into two patient groups (white vs non-white).

Subgroup Analysis 3:
We stratified appointments based on whether private insurance was used for each given appointment. Private insurance was used in 68.1% of all completed appointments.
The results of these analyses are summarized in eTable 6. These findings are consistent with our study's primary findings: Across all strata, VEText was associated with a decrease in waiting time, each incomplete booking was associated with additional waiting time, and VEText was associated with an exacerbation of delay from incomplete bookings.

Control Type Specific control variable Patient's sociodemographic profile
Age at the time of the appointment, and square of the patient's age.
Whether or not the patient used private insurance to supplement the financing of the appointment.

Seasonal / cyclical factors
Calendar month of the appointment, modelled as a categorical variable (levels: January, February, …, October).
Day of the week of the appointment, modelled as a categorical variable (levels: Sunday, Monday, …, Saturday).

Patient's utilization of VA health services
Total number of appointments made by that patient over the study period.
Total number of different medical classes that the patient had appointments with over the study period.

Clinic's congestion level
Total number of completed appointments in the same week.
Total number of booked appointments in the same week.
Total number of patients seen over the entire study period.

Individual facilities and appointment classes
Categorical variable indicating the VA health care facility for the appointment (130 levels in total, one for each facility).