Prediction Tools for Psychiatric Adverse Effects After Levetiracetam Prescription | Clinical Pharmacy and Pharmacology | JAMA Neurology | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Sensitivity and Specificity Trade-off for Predicting a Psychiatric Adverse Event After Prescription of Levetiracetam
Sensitivity and Specificity Trade-off for Predicting a Psychiatric Adverse Event After Prescription of Levetiracetam

Shown are results according to variations in the risk model probability threshold.

Figure 2.  Probability Gradient of Reporting a Psychiatric Sign, Symptom, or Disorder Within 2 Years of an Index Prescription for Levetiracetam According to the Number of Risk Factors (Female Sex, History of Depression, History of Anxiety, and Recreational Drug Use) Present
Probability Gradient of Reporting a Psychiatric Sign, Symptom, or Disorder Within 2 Years of an Index Prescription for Levetiracetam According to the Number of Risk Factors (Female Sex, History of Depression, History of Anxiety, and Recreational Drug Use) Present

Each bar represents the risk attributed to a unique combination of the 4 identified risk factors. For those with 2 or 3 risk factors, the probability tended to be higher for individuals with a history of depression or recreational drug use.

Figure 3.  Sensitivity and Specificity Trade-off for Predicting a Psychiatric Adverse Event After Prescription of Levetiracetam for Patients With No History of a Psychiatric Sign, Symptom, or Disorder
Sensitivity and Specificity Trade-off for Predicting a Psychiatric Adverse Event After Prescription of Levetiracetam for Patients With No History of a Psychiatric Sign, Symptom, or Disorder

Shown are results according to variations in the risk model probability threshold.

Table.  Demographic Characteristics of 1173 Patients With Incident Epilepsy Receiving Levetiracetam During Follow-up in The Health Improvement Network General Practice Database Stratified by Receipt of a Postprescription Code for a Psychiatric Sign, Symptom, or Disordera
Demographic Characteristics of 1173 Patients With Incident Epilepsy Receiving Levetiracetam During Follow-up in The Health Improvement Network General Practice Database Stratified by Receipt of a Postprescription Code for a Psychiatric Sign, Symptom, or Disordera
1.
Pickrell  WO, Lacey  AS, Thomas  RH, Lyons  RA, Smith  PE, Rees  MI.  Trends in the first antiepileptic drug prescribed for epilepsy between 2000 and 2010.  Seizure. 2014;23(1):77-80. doi:10.1016/j.seizure.2013.09.007PubMedGoogle ScholarCrossref
2.
Arif  H, Buchsbaum  R, Weintraub  D,  et al.  Comparison and predictors of rash associated with 15 antiepileptic drugs.  Neurology. 2007;68(20):1701-1709. doi:10.1212/01.wnl.0000261917.83337.dbPubMedGoogle ScholarCrossref
3.
Mawhinney  E, Craig  J, Morrow  J,  et al.  Levetiracetam in pregnancy: results from the UK and Ireland epilepsy and pregnancy registers  [published correction appears in Neurology. 2013;80(7):691].  Neurology. 2013;80(4):400-405. doi:10.1212/WNL.0b013e31827f0874PubMedGoogle ScholarCrossref
4.
Shallcross  R, Bromley  RL, Cheyne  CP,  et al; Liverpool and Manchester Neurodevelopment Group; UK Epilepsy and Pregnancy Register.  In utero exposure to levetiracetam vs valproate: development and language at 3 years of age.  Neurology. 2014;82(3):213-221. doi:10.1212/WNL.0000000000000030PubMedGoogle ScholarCrossref
5.
Schmidt  D, Schachter  SC.  Drug treatment of epilepsy in adults.  BMJ. 2014;348:g254. doi:10.1136/bmj.g254PubMedGoogle ScholarCrossref
6.
Weintraub  D, Buchsbaum  R, Resor  SR  Jr, Hirsch  LJ.  Psychiatric and behavioral side effects of the newer antiepileptic drugs in adults with epilepsy.  Epilepsy Behav. 2007;10(1):105-110. doi:10.1016/j.yebeh.2006.08.008PubMedGoogle ScholarCrossref
7.
Chen  Z, Lusicic  A, O’Brien  TJ, Velakoulis  D, Adams  SJ, Kwan  P.  Psychotic disorders induced by antiepileptic drugs in people with epilepsy.  Brain. 2016;139(Pt 10):2668-2678. doi:10.1093/brain/aww196PubMedGoogle ScholarCrossref
8.
French  JA, Staley  BA.  AED treatment through different ages: as our brains change, should our drug choices also?  Epilepsy Curr. 2012;12(suppl 3):22-27. doi:10.5698/1535-7511-12.4s.22PubMedGoogle ScholarCrossref
9.
UCL Institute of Epidemiology and Health Care. THIN database. https://www.ucl.ac.uk/iehc/research/primary-care-and-population-health/research/thin-database/database. Accessed December 13, 2018.
10.
Chisholm  J.  The Read clinical classification.  BMJ. 1990;300(6732):1092. doi:10.1136/bmj.300.6732.1092PubMedGoogle ScholarCrossref
11.
García Rodríguez  LA, Pérez Gutthann  S.  Use of the UK General Practice Research Database for pharmacoepidemiology.  Br J Clin Pharmacol. 1998;45(5):419-425. doi:10.1046/j.1365-2125.1998.00701.xPubMedGoogle ScholarCrossref
12.
 British National Formulary. 59th ed. London, United Kingdom: British Medical Association and Royal Pharmaceutical Society of Great Britain; 2009.
13.
Meeraus  WH, Petersen  I, Chin  RF, Knott  F, Gilbert  R.  Childhood epilepsy recorded in primary care in the UK.  Arch Dis Child. 2013;98(3):195-202. doi:10.1136/archdischild-2012-302237PubMedGoogle ScholarCrossref
14.
Fonferko-Shadrach  B, Lacey  AS, White  CP,  et al.  Validating epilepsy diagnoses in routinely collected data.  Seizure. 2017;52:195-198. doi:10.1016/j.seizure.2017.10.008PubMedGoogle ScholarCrossref
15.
Maguire  A, Blak  BT, Thompson  M.  The importance of defining periods of complete mortality reporting for research using automated data from primary care.  Pharmacoepidemiol Drug Saf. 2009;18(1):76-83. doi:10.1002/pds.1688PubMedGoogle ScholarCrossref
16.
Fitch  K, Bernstein  SJ, Aguilar  MD,  et al.  The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corp; 2001.
17.
Rubin  DB, Schenker  N.  Multiple imputation in health-care databases: an overview and some applications.  Stat Med. 1991;10(4):585-598. doi:10.1002/sim.4780100410PubMedGoogle ScholarCrossref
18.
Wlodzimirow  KA, Eslami  S, Chamuleau  RA, Nieuwoudt  M, Abu-Hanna  A.  Prediction of poor outcome in patients with acute liver failure-systematic review of prediction models.  PLoS One. 2012;7(12):e50952. doi:10.1371/journal.pone.0050952PubMedGoogle ScholarCrossref
19.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2PubMedGoogle ScholarCrossref
20.
Stata Statistical Software [computer program]. Release 13. College Station, TX: StataCorp LP; 2013.
21.
Pedregosa  F, Varoquaux  G, Gramfort  A,  et al.  Scikit-learn: machine learning in Python.  J Mach Learn Res. 2011;12:2825-2830.Google Scholar
22.
Mula  M, Trimble  MR, Yuen  A, Liu  RS, Sander  JW.  Psychiatric adverse events during levetiracetam therapy.  Neurology. 2003;61(5):704-706. doi:10.1212/01.WNL.0000078031.32904.0DPubMedGoogle ScholarCrossref
23.
Lee  K, Moon  J, Shin  J,  et al. Psychiatric adverse events after levetiracetam use in Korean adults: independent from serum drug level. In: Proceedings from the 2013 Annual Meeting of the American Epilepsy Society; December 5-9, 2013; Washington, DC. Abstract 2.142.
24.
Briggs  DE, French  JA.  Levetiracetam safety profiles and tolerability in epilepsy patients.  Expert Opin Drug Saf. 2004;3(5):415-424. doi:10.1517/14740338.3.5.415PubMedGoogle ScholarCrossref
25.
Brodtkorb  E, Klees  TM, Nakken  KO, Lossius  R, Johannessen  SI.  Levetiracetam in adult patients with and without learning disability: focus on behavioral adverse effects.  Epilepsy Behav. 2004;5(2):231-235. doi:10.1016/j.yebeh.2003.12.005PubMedGoogle ScholarCrossref
26.
Farooq  MU, Bhatt  A, Majid  A, Gupta  R, Khasnis  A, Kassab  MY.  Levetiracetam for managing neurologic and psychiatric disorders.  Am J Health Syst Pharm. 2009;66(6):541-561. doi:10.2146/ajhp070607PubMedGoogle ScholarCrossref
27.
Kaufman  KR, Bisen  V, Zimmerman  A, Tobia  A, Mani  R, Wong  S.  Apparent dose-dependent levetiracetam-induced de novo major depression with suicidal behavior.  Epilepsy Behav Case Rep. 2013;1:110-112. doi:10.1016/j.ebcr.2013.07.002PubMedGoogle ScholarCrossref
28.
Welty  TE, Gidal  BE, Ficker  DM, Privitera  MD.  Levetiracetam: a different approach to the pharmacotherapy of epilepsy.  Ann Pharmacother. 2002;36(2):296-304. doi:10.1345/aph.1A032PubMedGoogle ScholarCrossref
29.
Ben-Menachem  E, Gilland  E.  Efficacy and tolerability of levetiracetam during 1-year follow-up in patients with refractory epilepsy.  Seizure. 2003;12(3):131-135. doi:10.1016/S1059-1311(02)00251-0PubMedGoogle ScholarCrossref
30.
Morris  R, Carstairs  V.  Which deprivation? a comparison of selected deprivation indexes.  J Public Health Med. 1991;13(4):318-326.PubMedGoogle Scholar
31.
Josephson  CB, Engbers  JDT, Jette  N,  et al.  Prescription trends and psychiatric symptoms following first receipt of one of seven common antiepileptic drugs in general practice.  Epilepsy Behav. 2018;84:49-55. doi:10.1016/j.yebeh.2018.04.012PubMedGoogle ScholarCrossref
32.
Fiest  KM, Sauro  KM, Wiebe  S,  et al.  Prevalence and incidence of epilepsy: A systematic review and meta-analysis of international studies.  Neurology. 2017;88(3):296-303. doi:10.1212/WNL.0000000000003509PubMedGoogle ScholarCrossref
33.
Josephson  CB, Engbers  JD, Sajobi  TT,  et al.  An investigation into the psychosocial effects of the postictal state.  Neurology. 2016;86(8):723-730. doi:10.1212/WNL.0000000000002398PubMedGoogle ScholarCrossref
34.
Stefansson  SB, Olafsson  E, Hauser  WA.  Psychiatric morbidity in epilepsy: a case controlled study of adults receiving disability benefits.  J Neurol Neurosurg Psychiatry. 1998;64(2):238-241. doi:10.1136/jnnp.64.2.238PubMedGoogle ScholarCrossref
35.
Fiest  KM, Dykeman  J, Patten  SB,  et al.  Depression in epilepsy: a systematic review and meta-analysis.  Neurology. 2013;80(6):590-599. doi:10.1212/WNL.0b013e31827b1ae0PubMedGoogle ScholarCrossref
36.
Wiglusz  MS, Landowski  J, Cubała  WJ.  Prevalence of anxiety disorders in epilepsy.  Epilepsy Behav. 2018;79:1-3. doi:10.1016/j.yebeh.2017.11.025PubMedGoogle ScholarCrossref
37.
Clancy  MJ, Clarke  MC, Connor  DJ, Cannon  M, Cotter  DR.  The prevalence of psychosis in epilepsy: a systematic review and meta-analysis.  BMC Psychiatry. 2014;14:75. doi:10.1186/1471-244X-14-75PubMedGoogle ScholarCrossref
38.
Jehi  L, Yardi  R, Chagin  K,  et al.  Development and validation of nomograms to provide individualised predictions of seizure outcomes after epilepsy surgery: a retrospective analysis.  Lancet Neurol. 2015;14(3):283-290. doi:10.1016/S1474-4422(14)70325-4PubMedGoogle ScholarCrossref
39.
Lamberink  HJ, Otte  WM, Geerts  AT,  et al.  Individualised prediction model of seizure recurrence and long-term outcomes after withdrawal of antiepileptic drugs in seizure-free patients: a systematic review and individual participant data meta-analysis.  Lancet Neurol. 2017;16(7):523-531. doi:10.1016/S1474-4422(17)30114-XPubMedGoogle ScholarCrossref
40.
Rait  G, Walters  K, Griffin  M, Buszewicz  M, Petersen  I, Nazareth  I.  Recent trends in the incidence of recorded depression in primary care.  Br J Psychiatry. 2009;195(6):520-524. doi:10.1192/bjp.bp.108.058636PubMedGoogle ScholarCrossref
41.
Walters  K, Rait  G, Griffin  M, Buszewicz  M, Nazareth  I.  Recent trends in the incidence of anxiety diagnoses and symptoms in primary care.  PLoS One. 2012;7(8):e41670. doi:10.1371/journal.pone.0041670PubMedGoogle ScholarCrossref
Original Investigation
January 28, 2019

Prediction Tools for Psychiatric Adverse Effects After Levetiracetam Prescription

Author Affiliations
  • 1Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
  • 2Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Alberta, Canada
  • 3Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta, Canada
  • 4O’Brien Institute for Public Health, University of Calgary, Calgary, Alberta, Canada
  • 5Desid Labs Inc, Calgary, Alberta, Canada
  • 6Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, New York
  • 7Department of Psychiatry, University of Calgary, Calgary, Alberta, Canada
  • 8Department of Psychology, University of Calgary, Calgary, Alberta, Canada
  • 9Clinical Research Unit, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
JAMA Neurol. 2019;76(4):440-446. doi:10.1001/jamaneurol.2018.4561
Key Points

Question  Can routine clinical data be used to predict which patients with epilepsy will experience a psychiatric adverse effect from levetiracetam?

Findings  Among 1173 patients with epilepsy receiving levetiracetam in this open cohort study, 2 prediction models were created: one for the overall population and one for those without a history of a psychiatric sign, symptom, or disorder during the study period. The corresponding areas under the curve were 68% and 72%, respectively, and specificity was maximized using threshold cutoffs of 0.10 (full model) and 0.14 (second model); a score below these thresholds indicates safety of prescription.

Meaning  Levetiracetam has rapidly become a drug of first choice, and these models can be used to predict the risk of psychiatric adverse effects.

Abstract

Importance  Levetiracetam is a commonly used antiepileptic drug, yet psychiatric adverse effects are common and may lead to treatment discontinuation.

Objective  To derive prediction models to estimate the risk of psychiatric adverse effects from levetiracetam use.

Design, Setting, and Participants  Retrospective open cohort study. All patients meeting the case definition for epilepsy after the Acceptable Mortality Reporting date in The Health Improvement Network (THIN) database based in the United Kingdom (inclusive January 1, 2000, to May 31, 2012) who received a first-ever prescription for levetiracetam were included. Of 11 194 182 patients registered in THIN, this study identified 7400 presumed incident cases (66.1 cases per 100 000 persons) over a maximum of 12 years’ follow-up. The index date was when patients received their first prescription code for levetiracetam, and follow-up lasted 2 years or until an event, loss to follow-up, or censoring. The analyses were performed on April 22, 2018.

Exposure  A presumed first-ever prescription for levetiracetam.

Main Outcomes and Measures  The outcome of interest was a Read code for any psychiatric sign, symptom, or disorder as reached through consensus by 2 authors. This study used regression techniques to derive 2 prediction models, one for the overall population and one for those without a history of a psychiatric sign, symptom, or disorder during the study period.

Results  Among 1173 patients with epilepsy receiving levetiracetam, the overall median age was 39 (interquartile range, 25-56) years, and 590 (50.3%) were female. A total of 14.1% (165 of 1173) experienced a psychiatric symptom or disorder within 2 years of index prescription. The odds of reporting a psychiatric symptom were significantly elevated for women (odds ratio [OR], 1.41; 95% CI, 0.99-2.01; P = .05) and those with a preexposure history of higher social deprivation (OR, 1.15; 95% CI, 1.01-1.31; P = .03), depression (OR, 2.20; 95% CI, 1.49-3.24; P < .001), anxiety (OR, 1.74; 95% CI, 1.11-2.72; P = .02), or recreational drug use (OR, 2.02; 95% CI, 1.20-3.37; P = .008). The model performed well after stratified k = 5-fold cross-validation (area under the curve [AUC], 0.68; 95% CI, 0.58-0.79). There was a gradient in risk, with probabilities increasing from 8% for 0 risk factors to 11% to 17% for 1, 17% to 31% for 2, 30% to 42% for 3, and 49% when all risk factors were present. For those free of a preexposure psychiatric code, a second model performed comparably well after k = 5-fold cross-validation (AUC, 0.72; 95% CI, 0.54-0.90). Specificity was maximized using threshold cutoffs of 0.10 (full model) and 0.14 (second model); a score below these thresholds indicates safety of prescription.

Conclusions and Relevance  This study derived 2 simple models that predict the risk of a psychiatric adverse effect from levetiracetam. These algorithms can be used to guide prescription in clinical practice.

Introduction

Quiz Ref IDLevetiracetam is a commonly used antiepileptic drug (AED) in clinical practice. Its prescription has steadily increased since receiving approval in multiple jurisdictions in the early 2000s and has rapidly become a drug of choice for epilepsy.1 This is likely due to physician comfort related to its ease of use, efficacy, broad spectrum of action, low risks of idiosyncratic or life-threatening events2 or major congenital malformations,3,4 and lack of pharmacologic interactions.5 Levetiracetam can also be rapidly titrated in urgent situations, further establishing it as a first-line agent among neurologists and primary care physicians working in emergency settings.

However, a well-recognized complication is psychiatric adverse reactions, a potentially disruptive phenomenon that occurs in up to 16% of patients and frequently necessitates discontinuation.6,7 If a physician is able to predict who will develop a psychiatric adverse effect at an individual level, then personalized medicine can supplant trial-and-error approaches by selecting only patients at low risk for adverse effects. Despite this, predictive models have not been developed to date that can be used to identify persons with epilepsy at risk of psychiatric adverse events and hence guide prescription, titration, and counseling in this patient population.

Our objective was to develop such a rule using a combination of clinical expertise and data-driven processes. We chose to focus on adults because the adverse event profile of an AED can vary considerably between these populations.8 Using a data-driven, clinically informed approach with cross-validation techniques, we sought to generate predictive models guiding levetiracetam prescription that can be generalized to independent populations.

Methods
The Health Improvement Network

The Health Improvement Network (THIN) database is an electronic medical record (EMR) data platform based in the United Kingdom that consists of anonymized general practice (GP) patient records. All nonemergency specialist care in the United Kingdom requires GP registration. This data set contains GP records for a representative sample of approximately 5% of the national population.9 Medical events are coded using Read codes,10 and prescription data are classified according to the British National Formulary.11,12 This study was performed using THIN version 1205 (inclusive January 1, 2000, to May 31, 2012). The analyses were performed on April 22, 2018.

THIN received approval from the National Health Service South East Multi-centre Research Ethics Committee in 2003. All patients provide written informed consent. Ethics approval for this study was obtained both through the University of Calgary’s Conjoint Health Research Ethics Board and the Cegedim Strategic Data–Medical Research UK (CSD-MR UK) Scientific Review Committee in December 2015.

Study Population

We used a retrospective open cohort design for this study. A standard 5-year washout was applied starting at the date of patient enrollment (during which the patient could not receive any codes for epilepsy) to minimize the chance of inadvertently classifying prevalent epilepsy cases as incident. The epilepsy case definition for THIN requires either a single Read code for an epilepsy syndrome or 2 Read codes for symptoms of epilepsy plus 2 AED codes within 4 months.13 This case definition has 92% accuracy for detecting cases of pediatric epilepsy in THIN and has a high sensitivity and specificity (86% and 97%, respectively) in a similarly designed Welsh database (the Secure Anonymised Information Linkage [SAIL] Databank).13,14 Inclusion of patients occurred after their practice met the Acceptable Mortality Reporting date (the date when mortality reporting was considered complete) in THIN.15 All patients 18 years or older at epilepsy diagnosis who met these conditions were included in the analysis.

Exposure and Outcome

A prescription code for levetiracetam represented the exposure. This could occur at any point after the incident diagnosis as either monotherapy or adjunctive therapy. We defined the outcome of interest as a Read or Multilex therapeutic code for any psychiatric sign, symptom, or disorder, as defined through a consensus-driven process between 2 of us with expertise in epilepsy (C.B.J.) and psychiatry (S.B.P.) (eAppendix 1 in the Supplement).

Selection of Predictor Clinical Variables

We first developed a search strategy (eAppendix 2 in the Supplement) to interrogate MEDLINE and Embase for candidate predictors. Senior authors who are epileptologists with extensive clinical experience (N.J. and S.W.) provided additional expert opinion.

A modified Delphi process was then used for final variable selection. The 12-member panel comprised 7 adult epileptologists (C.B.J., N.J., S.S., Y.A.-K., P.F., N.P., and S.W.), 3 psychiatrists (S.B.P., A.M., and B.M.), and 2 clinical psychologists (S.M. and R.S.) from the University of Calgary, with a median total of 10.5 years (interquartile range [IQR], 4-25 years) of practice experience. Each participant completed a questionnaire rating all potential variables on a 5-point Likert-type scale (where 1 indicates the criterion was not very important at all and 5 indicates the criterion was very important). Panelists were also given space for additional handwritten comments. We assigned each panelist a random code and allotted 14 days to complete the first round of ranking. A follow-up email was sent on day 10.

Items with median ratings of 1 to 2 were considered irrelevant and were excluded, items scoring 3 were considered to be of uncertain relevance and were reevaluated in a second round, and items scoring 4 to 5 were used to build the final predictive model. Medians falling in intermediate ranges (ie, 2.5 and 3.5) were placed in the higher relevance category.16 Disagreement occurred when at least 4 panelists rated the item as irrelevant (1-2) and at least 4 panelists rated the same item as relevant (4-5).16

The second questionnaire was similar, although panelists were provided additional information regarding the frequency distribution of how each item was initially scored, along with a reminder of how they individually ranked the item. Questions related to items in which there was disagreement were reworded to address possible confounding by artifactual disagreement.16 Also included were extra items identified by panelists during the first round. To facilitate variable inclusion, we used a consensus-building process between 2 of us (C.B.J. and S.S.) to identify Read and Multilex codes to define each clinical feature of interest in THIN.

Statistical Analysis

This study used regression techniques to derive 2 prediction models, one for the overall population and one for those without a history of a psychiatric sign, symptom, or disorder during the study period. The index date (time zero) was that on which the patient received the first documented levetiracetam prescription (after meeting the epilepsy case definition). Patients were followed up for 2 years or until an event, loss to follow-up, or censoring. Data were fully available for all variables except levetiracetam daily dose (65% complete). There was no evidence that data were missing not at random; therefore, we used Rubin and Schenker’s multiple imputation to replace null values.17 All variables achieving a significance of P ≤ .05 in univariable analyses were further evaluated in multivariable logistic regression. The final model consisted of all significant variables at 2-sided P ≤ .05. We assessed model performance using the Brier score18 (a measure of the forecasting accuracy for probabilistic predictions), model discrimination using the mean area under the curve (AUC), calibration using the Hosmer-Lemeshow goodness-of-fit test, and generalizability using the mean AUC derived from stratified k = 5-fold cross-validation to account for outcome imbalance between groups.19 The predicted probability was calculated using 1 / [1 + exp (−risk score)], where the risk score is equal to the output of the multivariable logistic regression model. We mapped sensitivity and specificity according to the probability cutoffs. Because we cannot rule out recrudescence of a preexisting mental health disorder, we repeated the same process but excluded all patients with a pre–index date Read code for a psychiatric sign or symptom. All analyses were performed using Hive version 0.13.1 (The Apache Software Foundation), Stata version 13.0 (StataCorp LP),20 and Python version 3.2 (Python Software Foundation).21

Results
Variable Selection

Our search of MEDLINE (from 1946) and Embase (from 1974) yielded 136 articles, of which 103 remained after deduplication. Histories of febrile seizures,22 status epilepticus,22 longer duration of epilepsy,23 psychiatric comorbidities and behavioral issues,22-28 and cognitive impairment29 were associated with psychiatric adverse effects from levetiracetam use. Quiz Ref IDCoadministration of lamotrigine was associated with a protective effect.22 An additional 29 variables were recommended for inclusion after consultation with senior authors. Therefore, the first round of the Delphi process consisted of a questionnaire comprising 36 items (eAppendix 3 in the Supplement).

Consensus was ultimately achieved for 14 of 36 variables, of which 12 (85.7%) were considered relevant (eTable 1 in the Supplement). Uncertainty still remained for the residual 22 variables, and 13 extra items were identified through the first round. Therefore, the second round consisted of 35 items, of which 11 (31.4%) met consensus for inclusion (eTable 2 in the Supplement). Ultimately, 2 variables were subsequently excluded due to redundancy (history of axis I disorder) and challenges in phenotyping the variable in EMR data (multiple drug adverse effects). Therefore, we evaluated 21 variables (12 from the first round and 9 from the second round) for inclusion in the prediction models (Table).

Prediction Modeling

Of 11 194 182 patients registered in THIN, we identified 7400 presumed incident cases (66.1 cases per 100 000 persons) over a maximum of 12 years’ follow-up. Approximately 16% (n = 1173) received an incident prescription for levetiracetam during this period; the overall median age was 39 years (IQR, 25-56 years), and 590 (50.3%) were female. Psychiatric disorders like depression, anxiety, and psychosis were encountered in 22.3% (262 of 1173), 13.6% (160 of 1173), and 3.2% (37 of 1173), respectively. Of those receiving levetiracetam, 165 (14.1%) experienced an outcome of any psychiatric symptom or therapeutic code over 2 years of follow-up (Table).

The median time from prescription to receipt of a psychiatric code (not necessarily symptom onset) was 5.7 months (IQR, 2-11 months). Depression and behavioral issues were the most commonly reported adverse effects (eTable 3 in the Supplement). The incidence of psychiatric adverse events as a proportion of total new levetiracetam prescriptions by year peaked in 2004 (22.6% [7 of 31]) and then fluctuated between 8.0% (4 of 50) and 18.2% (19 of 104) over the next 8 years (eFigure in the Supplement).

Quiz Ref IDPatients reporting a psychiatric adverse effect were statistically more likely to have several characteristics. They were more likely to be female (58.8% [97 of 165] vs 48.9% [493 of 1008], P = .02); be of lower socioeconomic status according to the Townsend Index of Social Deprivation30 (median, 3 [IQR, 2-4] vs median, 3 [IQR, 2-4]; P = .002); have a history of medical and social conditions, including depression (42.4% [70 of 165] vs 19.0% [192 of 1008], P < .001), anxiety (25.5% [42 of 165] vs 11.7% [118 of 1008], P < .001), personality disorder (6.1% [10 of 165] vs 1.2% [12 of 1008], P < .001), suicidal ideation/suicide attempts (7.3% [12 of 165] vs 2.6% [26 of 1008], P = .002), and recreational drug use (18.2% [30 of 165] vs 6.4% [65 of 1008], P < .001); and have taken a psychotropic medication (26.1% [43 of 165] vs 17.4% [175 of 1008], P = .008).

When evaluating all significant variables in multivariable logistic regression, including somatic and psychiatric signs, symptoms, and disorders, the odds of reporting a psychiatric symptom or treatment code within 2 years of persistent levetiracetam use were elevated for some patients. These included those with the following characteristics: female sex (odds ratio [OR], 1.41; 95% CI, 0.99-2.01; P = .05), increasing social deprivation (OR, 1.15; 95% CI, 1.01-1.31; P = .03), depression (OR, 2.20; 95% CI, 1.49-3.24; P < .001), or anxiety (OR, 1.74; 95% CI, 1.11-2.72; P = .02), and recreational drug use (OR, 2.02; 95% CI, 1.20-3.37; P = .008).

Although it is valuable to know that low socioeconomic status increases the odds of reporting a psychiatric symptom or disorder after levetiracetam prescription, a decision was made to exclude the Townsend Index of Social Deprivation from the final model because it would be challenging to assign this score in clinic. Quiz Ref IDHence, the final model was as follows: risk score = −2.34 + 0.27 × (female sex) + 0.82 × (history of depression) + 0.47 × (history of anxiety) + 0.74 × (history of recreational drug use).

There was no evidence of multicollinearity, with variance inflation factors ranging from 1.15 (history of recreational drug use) to 1.42 (history of depression). The model performed well, with a Brier score of 0.11. It had moderate discriminative capacity and appeared generalizable after stratified k = 5-fold cross-validation (AUC, 0.68; 95% CI, 0.58-0.79). The sensitivity and specificity varied, but high specificity (83%) was achieved at a probability cutoff of 0.10 (Figure 1). There was no evidence of poor calibration based on the Hosmer-Lemeshow goodness-of-fit test (P = .29; 10 groups). A gradient was revealed in which those with increasing numbers of risk factors were at incrementally greater risk of a psychiatric outcome (Figure 2). Baseline risk for those with no risk factors was 8%, the risk for those with 1 risk factor ranged from 11% to 17%, the risk for those with 2 risk factors ranged from 17% to 31%, the risk for those with 3 risk factors ranged from 30% to 42%, and the risk for those with all 4 risk factors was 49% (Figure 2 and eTable 4 in the Supplement). For each stratum, the presence of depression and recreational drug use, singularly or in combination, was consistently associated with higher risk.

For those with no documented pre–index date history of a mental health sign, symptom, or disorder (710 of 1173 [60.5%]), 69 of 710 (9.7%) received a code for a psychiatric event within 2 years of levetiracetam prescription. The following model was generated: risk score = −3.83 + 0.013 × (age) + 0.89 × (female sex) + 1.16 × (recreational drug use) + 0.0003 × (levetiracetam daily dose).

There was no evidence of multicollinearity (variance inflation factors range, 1.03-2.83) or poor calibration based on the Hosmer-Lemeshow goodness-of-fit test (P = .18; 10 groups). The model performed well overall (Brier score, 0.09) and when using a threshold of 0.14 had a similar specificity of 83% (Figure 3). Discriminatory capacity and generalizability, as determined through stratified k = 5-fold cross-validation, was similar to the primary model (AUC, 0.72; 95% CI, 0.54-0.90).

Discussion

We were able to derive 2 prediction models for the risk of a psychiatric sign or symptom after a first-ever prescription for levetiracetam in a general population of patients with presumed incident epilepsy. When using a threshold value of 0.10 (primary model) or 0.14 (for those lacking a premorbid psychiatric history) to guide treatment (a score below which indicates safety of prescription), the prediction models have a high specificity for predicting those with a psychiatric outcome, yielding few false-positive results. If validated in prospective cohorts, these models could be useful at the point of care for a broad spectrum of patients with epilepsy seen in GP and epilepsy clinics alike.

The discovered incremental probability of reporting a psychiatric sign can help generate an index of suspicion to counsel patients. In addition, because there is evidence that preexposure psychiatric symptoms or disorders may predict the emergence of similar issues after initiation of many AEDs,31 the second prediction model was generated to rule out any confounding consequences of these variables. Our study also highlights the independent influence that concurrent low socioeconomic status and recreational drug use exert on this association. Patients should not be denied access to medications because of their socioeconomic status, although knowledge of this interaction indicates closer monitoring may be required.

Our study benefited from a clinically informed approach using a modified Delphi panel composed of experts in the field with varied but extensive interest in epilepsy, AEDs, and neuropsychiatry. Our definitions of exposure and outcome also performed well when compared with published frequency estimates; for instance, the incidence proportion of epilepsy in our study (66.1 cases per 100 000 persons) is comparable to that reported in the literature.32 Likewise, the demographics and social habits mirrored what is expected based on prior studies.33,34 The exposures also appeared to be reliably coded; the proportions herein meeting case definitions for preexposure depression (22.3%),35 anxiety (13.6%),36 and mania/psychosis (3.2%)37 were consistent with the literature. Using EMR data meant variables were derived from routine clinic visits, implying direct relevance to everyday practice.

Limitations

There are potential limitations to our work. Quiz Ref IDWe were unable to extract informative indexes of seizure type, frequency, and severity and epilepsy type. Although we controlled for levetiracetam daily dose and concomitant AED use, it is impossible to extract measures of medication adherence. Inclusion of these variables could further improve model performance. Despite this, the discriminative ability of our model was similar to that published for nomograms predicting outcome after epilepsy surgery38 and AED withdrawal.39 General practitioners may not have been as attuned to the risk of psychiatric symptoms related to levetiracetam use as epileptologists; however, reassuringly, the proportion reporting the outcome of interest (14.1% [165 of 1173]) is consistent with the published literature.6 A critical advantage to this method is that expectation bias is circumvented because GPs would not be prone to anticipating the outcome of interest, have a lower threshold for diagnosing it, or be apt to treat those taking levetiracetam different than other patients. Therefore, by these means, we have mitigated conservative estimates of risk. We cannot exclude the possibility of underreporting of psychiatric events in primary care data repositories40,41 because formal validation has yet to be undertaken. This likely constitutes nondifferential misclassification bias in which the overall effect estimates are diluted by the equal reluctance of both exposed and unexposed individuals to report psychiatric symptoms due to a myriad of factors, including fear of stigma. This would be anticipated to produce conservative estimates of association. Finally, this issue may have resulted in the inadvertent inclusion of patients with false-negative results in our sensitivity analysis, where we excluded all those with any preexposure code for a psychiatric sign, symptom, or disorder. However, under these circumstances, this would be expected to result in a higher than anticipated rate of postexposure psychiatric adverse effects in the sensitivity cohort, who presumably lacked the risk factor of a premorbid psychiatric disorder. Yet, this was not the case (14.1% [165 of 1173] in the full cohort and 9.7% [69 of 710] in those lacking a preexposure code). Ideally, these definitions will undergo formal validation, including measures of sensitivity and specificity, in EMR settings.

Model validation should have a robust influence on the provision of personalized medicine because levetiracetam constitutes an ideal first-choice AED if we are able to confidently select those patients who are at low risk for psychiatric adverse effects. Although beyond the scope of this project, similar methods and data sources can be used to evaluate other broad-spectrum AEDs with idiosyncratic adverse effects, thus enhancing confidence in their use in GP. This is integral because these same risk factors may be equally applicable in predicting the risk of psychiatric adverse events for other AEDs. In addition, although the rule can be used to guide prescription of levetiracetam, risk scores that exceed the threshold do not necessarily mean alternate AEDs would result in better seizure outcomes. Rather, patient preferences, and their attendant comfort with the risk of adverse events, should be evaluated on an individual basis. Finally, caution is warranted in interpreting causation because machine learning regression techniques are designed for predictive rather than inferential purposes.

Conclusions

One of the benefits of the tools described herein is that they can be easily applied in clinical settings. For instance, a female patient with depression would have a risk score of −1.25 (ie, risk score = −2.34 + 0.27 + 0.82). When incorporated into the algorithm, the patient’s risk of a psychiatric adverse event would be 22% over 2 years: 1 / [1 + exp (−1.25)]. This exceeds the threshold of 0.10 and suggests she would be at risk. In addition, the gradients of risk discriminate populations of patients based on known and newly identified factors. The estimates of generalizability using a stratified k = 5-fold cross-validation are encouraging. This is an important first step toward generating empirical, efficient, and practical prediction models with direct application in clinical settings.

Back to top
Article Information

Accepted for Publication: November 8, 2018.

Published Online: January 28, 2019. doi:10.1001/jamaneurol.2018.4561

Correction: This article was corrected on March 18, 2019, to fix a typographical error in the second risk score model in the Prediction Modeling subsection of the Results.

Corresponding Author: Colin B. Josephson, MD, MSc, FRCPC, Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Foothills Medical Centre, 1403 29th St NW, Calgary, AB T2N 2T9, Canada (cbjoseph@ucalgary.ca).

Author Contributions: Dr Josephson had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Josephson, Engbers, Jette, Macrodimitris, Wiebe.

Acquisition, analysis, or interpretation of data: Josephson, Engbers, Jette, Patten, Singh, Sajobi, Marshall, Agha-Khani, Federico, Mackie, McLane, Pillay, Sharma, Wiebe.

Drafting of the manuscript: Josephson, Engbers, Agha-Khani, Sharma.

Critical revision of the manuscript for important intellectual content: Josephson, Jette, Patten, Singh, Sajobi, Marshall, Federico, Mackie, Macrodimitris, McLane, Pillay, Wiebe.

Statistical analysis: Josephson, Engbers, Sajobi.

Administrative, technical, or material support: Engbers, Pillay.

Supervision: Jette, Marshall, Macrodimitris, Wiebe.

Conflict of Interest Disclosures: Dr Josephson reported being the recipient of the American Brain Foundation/American Epilepsy Society/Epilepsy Foundation/American Academy of Neurology Susan S. Spencer Clinical Research Training Fellowship in Epilepsy and of an Alberta Innovates Health Solutions Clinician Fellowship. Dr Engbers reported being supported by an Alberta Innovates Health Solutions Postdoctoral Fellowship. Dr Jette reported being the holder of a Canada Research Chair Tier 2 in Neurological Health Services Research and reported receiving operating grants (paid to her institution) from the Canadian Institutes of Health Research, National Institute of Neurological Disorders and Stroke of the US National Institutes of Health, Alberta Health, and the University of Calgary Cumming School of Medicine. Dr Sajobi reported being supported by an M.S.I. Foundation New Investigator Grant. Dr Marshall reported being supported by a Canada Research Chair (Health Services and Systems Research) and the Arthur J. E. Child Chair in Rheumatology Outcomes Research and reported receiving operating grants (paid to the University of Calgary) from the Canadian Institutes of Health Research, Alberta Innovates Health Solutions, EuroQoL Research Foundation, Colorectal Cancer Association of Canada, Genome Canada, Canadian Initiative for Outcomes in Rheumatology cAre (CIORA), and the University of Calgary Cumming School of Medicine. Dr Wiebe reported being the holder of the Hopewell Professorship of Clinical Neurosciences Research at the University of Calgary.

References
1.
Pickrell  WO, Lacey  AS, Thomas  RH, Lyons  RA, Smith  PE, Rees  MI.  Trends in the first antiepileptic drug prescribed for epilepsy between 2000 and 2010.  Seizure. 2014;23(1):77-80. doi:10.1016/j.seizure.2013.09.007PubMedGoogle ScholarCrossref
2.
Arif  H, Buchsbaum  R, Weintraub  D,  et al.  Comparison and predictors of rash associated with 15 antiepileptic drugs.  Neurology. 2007;68(20):1701-1709. doi:10.1212/01.wnl.0000261917.83337.dbPubMedGoogle ScholarCrossref
3.
Mawhinney  E, Craig  J, Morrow  J,  et al.  Levetiracetam in pregnancy: results from the UK and Ireland epilepsy and pregnancy registers  [published correction appears in Neurology. 2013;80(7):691].  Neurology. 2013;80(4):400-405. doi:10.1212/WNL.0b013e31827f0874PubMedGoogle ScholarCrossref
4.
Shallcross  R, Bromley  RL, Cheyne  CP,  et al; Liverpool and Manchester Neurodevelopment Group; UK Epilepsy and Pregnancy Register.  In utero exposure to levetiracetam vs valproate: development and language at 3 years of age.  Neurology. 2014;82(3):213-221. doi:10.1212/WNL.0000000000000030PubMedGoogle ScholarCrossref
5.
Schmidt  D, Schachter  SC.  Drug treatment of epilepsy in adults.  BMJ. 2014;348:g254. doi:10.1136/bmj.g254PubMedGoogle ScholarCrossref
6.
Weintraub  D, Buchsbaum  R, Resor  SR  Jr, Hirsch  LJ.  Psychiatric and behavioral side effects of the newer antiepileptic drugs in adults with epilepsy.  Epilepsy Behav. 2007;10(1):105-110. doi:10.1016/j.yebeh.2006.08.008PubMedGoogle ScholarCrossref
7.
Chen  Z, Lusicic  A, O’Brien  TJ, Velakoulis  D, Adams  SJ, Kwan  P.  Psychotic disorders induced by antiepileptic drugs in people with epilepsy.  Brain. 2016;139(Pt 10):2668-2678. doi:10.1093/brain/aww196PubMedGoogle ScholarCrossref
8.
French  JA, Staley  BA.  AED treatment through different ages: as our brains change, should our drug choices also?  Epilepsy Curr. 2012;12(suppl 3):22-27. doi:10.5698/1535-7511-12.4s.22PubMedGoogle ScholarCrossref
9.
UCL Institute of Epidemiology and Health Care. THIN database. https://www.ucl.ac.uk/iehc/research/primary-care-and-population-health/research/thin-database/database. Accessed December 13, 2018.
10.
Chisholm  J.  The Read clinical classification.  BMJ. 1990;300(6732):1092. doi:10.1136/bmj.300.6732.1092PubMedGoogle ScholarCrossref
11.
García Rodríguez  LA, Pérez Gutthann  S.  Use of the UK General Practice Research Database for pharmacoepidemiology.  Br J Clin Pharmacol. 1998;45(5):419-425. doi:10.1046/j.1365-2125.1998.00701.xPubMedGoogle ScholarCrossref
12.
 British National Formulary. 59th ed. London, United Kingdom: British Medical Association and Royal Pharmaceutical Society of Great Britain; 2009.
13.
Meeraus  WH, Petersen  I, Chin  RF, Knott  F, Gilbert  R.  Childhood epilepsy recorded in primary care in the UK.  Arch Dis Child. 2013;98(3):195-202. doi:10.1136/archdischild-2012-302237PubMedGoogle ScholarCrossref
14.
Fonferko-Shadrach  B, Lacey  AS, White  CP,  et al.  Validating epilepsy diagnoses in routinely collected data.  Seizure. 2017;52:195-198. doi:10.1016/j.seizure.2017.10.008PubMedGoogle ScholarCrossref
15.
Maguire  A, Blak  BT, Thompson  M.  The importance of defining periods of complete mortality reporting for research using automated data from primary care.  Pharmacoepidemiol Drug Saf. 2009;18(1):76-83. doi:10.1002/pds.1688PubMedGoogle ScholarCrossref
16.
Fitch  K, Bernstein  SJ, Aguilar  MD,  et al.  The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corp; 2001.
17.
Rubin  DB, Schenker  N.  Multiple imputation in health-care databases: an overview and some applications.  Stat Med. 1991;10(4):585-598. doi:10.1002/sim.4780100410PubMedGoogle ScholarCrossref
18.
Wlodzimirow  KA, Eslami  S, Chamuleau  RA, Nieuwoudt  M, Abu-Hanna  A.  Prediction of poor outcome in patients with acute liver failure-systematic review of prediction models.  PLoS One. 2012;7(12):e50952. doi:10.1371/journal.pone.0050952PubMedGoogle ScholarCrossref
19.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2PubMedGoogle ScholarCrossref
20.
Stata Statistical Software [computer program]. Release 13. College Station, TX: StataCorp LP; 2013.
21.
Pedregosa  F, Varoquaux  G, Gramfort  A,  et al.  Scikit-learn: machine learning in Python.  J Mach Learn Res. 2011;12:2825-2830.Google Scholar
22.
Mula  M, Trimble  MR, Yuen  A, Liu  RS, Sander  JW.  Psychiatric adverse events during levetiracetam therapy.  Neurology. 2003;61(5):704-706. doi:10.1212/01.WNL.0000078031.32904.0DPubMedGoogle ScholarCrossref
23.
Lee  K, Moon  J, Shin  J,  et al. Psychiatric adverse events after levetiracetam use in Korean adults: independent from serum drug level. In: Proceedings from the 2013 Annual Meeting of the American Epilepsy Society; December 5-9, 2013; Washington, DC. Abstract 2.142.
24.
Briggs  DE, French  JA.  Levetiracetam safety profiles and tolerability in epilepsy patients.  Expert Opin Drug Saf. 2004;3(5):415-424. doi:10.1517/14740338.3.5.415PubMedGoogle ScholarCrossref
25.
Brodtkorb  E, Klees  TM, Nakken  KO, Lossius  R, Johannessen  SI.  Levetiracetam in adult patients with and without learning disability: focus on behavioral adverse effects.  Epilepsy Behav. 2004;5(2):231-235. doi:10.1016/j.yebeh.2003.12.005PubMedGoogle ScholarCrossref
26.
Farooq  MU, Bhatt  A, Majid  A, Gupta  R, Khasnis  A, Kassab  MY.  Levetiracetam for managing neurologic and psychiatric disorders.  Am J Health Syst Pharm. 2009;66(6):541-561. doi:10.2146/ajhp070607PubMedGoogle ScholarCrossref
27.
Kaufman  KR, Bisen  V, Zimmerman  A, Tobia  A, Mani  R, Wong  S.  Apparent dose-dependent levetiracetam-induced de novo major depression with suicidal behavior.  Epilepsy Behav Case Rep. 2013;1:110-112. doi:10.1016/j.ebcr.2013.07.002PubMedGoogle ScholarCrossref
28.
Welty  TE, Gidal  BE, Ficker  DM, Privitera  MD.  Levetiracetam: a different approach to the pharmacotherapy of epilepsy.  Ann Pharmacother. 2002;36(2):296-304. doi:10.1345/aph.1A032PubMedGoogle ScholarCrossref
29.
Ben-Menachem  E, Gilland  E.  Efficacy and tolerability of levetiracetam during 1-year follow-up in patients with refractory epilepsy.  Seizure. 2003;12(3):131-135. doi:10.1016/S1059-1311(02)00251-0PubMedGoogle ScholarCrossref
30.
Morris  R, Carstairs  V.  Which deprivation? a comparison of selected deprivation indexes.  J Public Health Med. 1991;13(4):318-326.PubMedGoogle Scholar
31.
Josephson  CB, Engbers  JDT, Jette  N,  et al.  Prescription trends and psychiatric symptoms following first receipt of one of seven common antiepileptic drugs in general practice.  Epilepsy Behav. 2018;84:49-55. doi:10.1016/j.yebeh.2018.04.012PubMedGoogle ScholarCrossref
32.
Fiest  KM, Sauro  KM, Wiebe  S,  et al.  Prevalence and incidence of epilepsy: A systematic review and meta-analysis of international studies.  Neurology. 2017;88(3):296-303. doi:10.1212/WNL.0000000000003509PubMedGoogle ScholarCrossref
33.
Josephson  CB, Engbers  JD, Sajobi  TT,  et al.  An investigation into the psychosocial effects of the postictal state.  Neurology. 2016;86(8):723-730. doi:10.1212/WNL.0000000000002398PubMedGoogle ScholarCrossref
34.
Stefansson  SB, Olafsson  E, Hauser  WA.  Psychiatric morbidity in epilepsy: a case controlled study of adults receiving disability benefits.  J Neurol Neurosurg Psychiatry. 1998;64(2):238-241. doi:10.1136/jnnp.64.2.238PubMedGoogle ScholarCrossref
35.
Fiest  KM, Dykeman  J, Patten  SB,  et al.  Depression in epilepsy: a systematic review and meta-analysis.  Neurology. 2013;80(6):590-599. doi:10.1212/WNL.0b013e31827b1ae0PubMedGoogle ScholarCrossref
36.
Wiglusz  MS, Landowski  J, Cubała  WJ.  Prevalence of anxiety disorders in epilepsy.  Epilepsy Behav. 2018;79:1-3. doi:10.1016/j.yebeh.2017.11.025PubMedGoogle ScholarCrossref
37.
Clancy  MJ, Clarke  MC, Connor  DJ, Cannon  M, Cotter  DR.  The prevalence of psychosis in epilepsy: a systematic review and meta-analysis.  BMC Psychiatry. 2014;14:75. doi:10.1186/1471-244X-14-75PubMedGoogle ScholarCrossref
38.
Jehi  L, Yardi  R, Chagin  K,  et al.  Development and validation of nomograms to provide individualised predictions of seizure outcomes after epilepsy surgery: a retrospective analysis.  Lancet Neurol. 2015;14(3):283-290. doi:10.1016/S1474-4422(14)70325-4PubMedGoogle ScholarCrossref
39.
Lamberink  HJ, Otte  WM, Geerts  AT,  et al.  Individualised prediction model of seizure recurrence and long-term outcomes after withdrawal of antiepileptic drugs in seizure-free patients: a systematic review and individual participant data meta-analysis.  Lancet Neurol. 2017;16(7):523-531. doi:10.1016/S1474-4422(17)30114-XPubMedGoogle ScholarCrossref
40.
Rait  G, Walters  K, Griffin  M, Buszewicz  M, Petersen  I, Nazareth  I.  Recent trends in the incidence of recorded depression in primary care.  Br J Psychiatry. 2009;195(6):520-524. doi:10.1192/bjp.bp.108.058636PubMedGoogle ScholarCrossref
41.
Walters  K, Rait  G, Griffin  M, Buszewicz  M, Nazareth  I.  Recent trends in the incidence of anxiety diagnoses and symptoms in primary care.  PLoS One. 2012;7(8):e41670. doi:10.1371/journal.pone.0041670PubMedGoogle ScholarCrossref
×