Psychosocial Morbidity: The Economic Burden in a Pediatric Health Maintenance Organization Sample | Geriatrics | JAMA Pediatrics | JAMA Network
[Skip to Navigation]
Sign In
March 2000

Psychosocial Morbidity: The Economic Burden in a Pediatric Health Maintenance Organization Sample

Author Affiliations

From the Department of Psychiatry, Kaiser Permanente Medical Group, San Jose, Calif (Drs Bernal, Bendell Estroff, and Aboudarham and Mr Keller); and the Child Psychiatry Service, Massachusetts General Hospital, and the Department of Psychiatry, Harvard Medical School, Boston, Mass (Drs Murphy and Jellinek).

Arch Pediatr Adolesc Med. 2000;154(3):261-266. doi:10.1001/archpedi.154.3.261

Objectives  To evaluate psychosocial morbidity in pediatric primary care and to determine displaced health care utilization.

Design and Setting  A cross-sectional sample of parent-child dyads was screened using the Pediatric Symptom Checklist (PSC) at 6 pediatric sites of a health maintenance organization (HMO). Cost and utilization data were retrieved from regional databases for this sample.

Participants  Parent-child dyads from an HMO in northern California (N = 1840). The children ranged in age from 2 to 18 years.

Results  In all, 13.0% of children exhibited psychosocial dysfunction. The rate of children's chronic illness was 18.4%. Multiple regression analyses measured utilization and cost of health and psychiatric care for the selected population for the previous year; the average log cost of health care per child was $393. The average health care cost for children with anxious, depressed symptoms was $805. Chronically ill children were the highest utilizers of health care, with an average log cost of $1138. When psychosocial dysfunction was present, regression models showed that health care spending was highest for young children.

Conclusions  Health care utilization was higher for children with psychosocial morbidity, was higher among younger children, and decreased with age as psychiatric costs progressively increased.

MANAGED CARE attempts to prioritize allocation of resources in health care delivery systems. Pediatricians are an important first resource for parents with concerns about their children's behavioral problems.1,2 Despite epidemiologic data suggesting that psychosocial morbidity is the most prevalent childhood problem (14% of children), only about 50% of these children are identified by their primary care physicians; once identified, only a small fraction receive appropriate mental health treatment.3-7 Beyond these constraints, pediatricians do not receive adequate training concerning psychosocial problems, may be hesitant to attach stigmatizing labels to children, do not have enough time during brief visits to address psychosocial needs, and may have limited access to mental health services.3,7,8

The economic implications of pediatric psychosocial morbidity within the general health care sectors have recently become a focus of empirical inquiry.9-11 Health care utilization studies suggest that adults with psychiatric disorders average twice as many visits to their primary care providers as those without them12; however, the findings for children and adolescents are less clear. Chronically ill children have more complex physical and mental health care profiles than healthy children and also have very high rates of medical utilization.13-15 Previous studies, although limited by small sample sizes and a narrow focus, suggested that pediatric psychopathologic processes were associated with increased general and subspecialty health care utilization.16-22 Known risk factors include parenting or family conflict; single-parent households were associated with high health care utilization.17-19,21-24 In addition, mothers' distress and health-related attitudes contributed to higher rates of pediatric health care utilization.25 Recently, Murphy and associates23 found that psychosocial morbidity was highly associated with health care utilization in a large, nationally representative pediatric population. Similarly, Zuckerman et al26 demonstrated that the most significant predictor of medical utilization in children was parent-reported behavior problems. Both of these studies relied on retrospective parental report of health care utilization rather than an objective measure.

The implementation of clinical data-tracking systems within health maintenance organizations (HMOs) provides a centralized and accessible measurement of cost for large patient populations and can objectively document health care utilization. Screening measures focusing on functional impairment have been suggested as an expedient method of establishing baseline rates of mental health morbidity for both parents and children who are at high psychosocial risk.5,27-30

In accord with previous studies of adult populations, we hypothesized that pediatric psychosocial morbidity would show an independent effect on increased health care utilization and costs.11,12,31 The effects of sociodemographic characteristics on health care utilization and costs were examined. Each parent and child's health care visit and the consequent medical costs during the year preceding and 6 months after screening were measured using the HMO's centralized clinical data-tracking system. With these data, baseline rates of psychosocial morbidity within the HMO's pediatric population were established and the economic burden within the sample was calculated using a cost analysis based on multiple linear regression.

Subjects and methods


The study was conducted at an HMO composed of 28 facilities in a 300-mile radius in northern California. Approximately 28% of 2.5 million members were younger than 18 years. Six facilities were used and all pediatricians and pediatric nurse practitioners (N = 185) participated; other primary care departments were not included. All types of urgent and routine pediatric visits were sampled.

All pediatrician schedules were used to recruit 300 parent-child dyads from each site in a rotating pattern. All children aged 2 to 18 years accompanied by a parent were eligible; adolescents without an accompanying parent were excluded. In all, 2475 dyads were approached and 1840 dyads were enrolled in the study. Primary reasons for refusal to participate were lack of time or interest. There were no statistically significant demographic differences between participants and nonparticipants.


Informed consent was obtained only from the parent. Data were collected for 4 hours a day across times, pediatric practices, and appointment types on any given day or evening over 4 months. Participants were enrolled only once during the data-collection period. All parents of children who scored above the clinical cutoff on the Pediatric Symptom Checklist (PSC) received a letter recommending appropriate follow-up services at the HMO's psychiatry department.


Pediatric Symptom Checklist

The PSC is a 35-item parent-completed checklist of children's emotional and behavioral problems associated with dysfunction.32-38 Item scores are added and the total score is recorded as a dichotomous variable indicating psychosocial impairment. Positive cutoff scores of 28 or higher for school-aged children (aged 6-18 years), and of 24 or higher for 2- to 5-year-old children were used. Cutoff scores were derived from receiver operator characteristic analyses in previous research comparing the performance of the PSC with the Children's Behavior Checklist for detecting psychosocial impairment.29,32-35 The PSC cutoff scores that provided the highest average sensitivity and specificity and the highest κ statistic when compared with Children's Behavior Checklist classification were selected. The PSC has been validated for diverse populations.35 The reliability and validity for screening 4- to 5-year-olds for English-speaking and Spanish-speaking children were also adequate.35

Physician Recognition of Children's Psychosocial Problems

Immediately after the index visit, physicians, without knowledge of PSC scores, documented the presence or absence of behavioral problems for the child on a brief questionnaire.

Chronic Illness

The presence of chronic illness was determined by parental report and by medical record review. In all, 14.2% of parents reported that their child was chronically ill at the time of screening.

Medical record review was based on inclusion criteria from ambulatory diagnostic groups.36 Records were reviewed to confirm the presence or absence of diagnoses in any patient who had been hospitalized, who was classified by parental report as chronically ill, or who had 5 or more medical visits in the previous year.15,17 There was consensus between 2 raters for these decisions. Since 18.6% of children were classified as chronically ill according to the medical records review, the report of chronic illness is based on these criteria.

Psychosocial Morbidity

Two factors of the PSC were extracted using principal components factor analysis with varimax rotation. Consistent with previous research, the 2 factors were grouped into symptoms of an internalizing nature (ie, depression and anxiety) and externalizing symptoms (ie, defiance and aggression). Separate factor analyses were run for younger and older age groups (aged 2-5 and 6-19 years), with the same results (Table 1). For purposes of subsequent analyses, the values for all externalizer items were added and the values for all internalizer items were added and multiplied by a ratio of 7:6 for calibration. All children who scored positively on the PSC and scored higher on the internalizer indicator were classified as internalizers, while all PSC-positive children who scored higher on the externalizing indicator were classified as externalizers. Children who had equal scores on both indicators (n = 7) were eliminated from the data analysis.

Table 1. 
Internalizing and Externalizing Indicators of the Pediatric Symptom Checklist (PSC) Derived From Factor Analysis With Varimax Rotation
Internalizing and Externalizing Indicators of the Pediatric Symptom Checklist (PSC) Derived From Factor Analysis With Varimax Rotation

Health Care Utilization

Utilization data for the study were extracted from the HMO automated systems that record all clinical costs, including hospitalizations, outpatient registrations, and pharmacy, laboratory, and radiological procedures for the child and the accompanying parent. Cost information from the general ledger was allocated using the Cost Management Information System (The Permanente Medical Group, San Francisco, Calif), purchased cost-accounting software that integrates cost and utilization. Within the system, the costs by department, medical center, patient, and service were allocated. For example, a routine 10-minute pediatric visit costs $65. The actual expenses for each service department were assigned relative values, which were used as weights for distributing the costs expensed in the department to the total volume of services provided therein. The resulting average unit cost was multiplied by the patient's actual units of service during a given visit to tabulate the direct costs of the encounter. Overhead costs were allocated to each unit of service via a step-down method. All program costs equivalent to the costs of "insurance-related" services, such as marketing, membership accounting, and related administrative costs, were excluded. As utilization accumulated for the study period under consideration, the costs of visits were aggregated. The total cost for each patient included the costs of emergency department care, primary care, specialty care, nursing, operating and recovery room care, and surgery, as well as associated radiology, pharmacy, and laboratory costs.

We also included authorized charges incurred for out-of-network services. Alternative and unauthorized medical care obtained outside the HMO network services were not included in the cost analysis. Costs of medical services and psychiatric services are reported separately and added as total health care costs.

Data analysis

Depending on the type of variables (continuous or discrete), Pearson r correlation or χ2 analyses were performed to evaluate associations among independent variables and between dependent and independent variables, as well as to identify potential confounding variables. Linear regression models were built to predict the utilization and cost as a function of being an externalizer (vs not), being an internalizer (vs not), education, and desire for additional services for the child. Two interaction terms were included: age and internalizer and age and externalizer. Socioeconomic status variables were included as covariates based on whether the correlations and χ2 analyses had shown them to be associated with the predictor variables. The age of the child and chronic illness were found to be associated with outcome variables in the analysis of variance. Thus, they were used as covariates in subsequent regression models. The variables were entered in the model in a stepwise method.37

Utilization and costs were aggregated for the period of 12 months prior to survey, for 6 months after the survey, for the calendar year of survey completion, and for 18 months, spanning from 12 months prior to 6 months after the survey. For each period, aggregated costs were considered as they were in one set of models and truncated to 3 SDs above the mean for another set of models. In each set, the model was run first for the entire sample with the indication of chronic illness (vs not) as a predictor, then separately for children who were and were not chronically ill.


Sociodemographic and psychosocial screening data are presented on the full sample (N = 1840) in Table 2. Utilization data were not available for 191 parent-child dyads because of partial enrollment (under 6 months) in the HMO within the 18-month study period. Thus, health care utilization and cost-effectiveness multiple regression analyses were completed using a sample of 1649 parent-child dyads. Subjects excluded from the multiple regression analyses did not differ from the rest of the sample on any demographic variables.

Table 2. 
Sociodemographic Characteristics of the Sample (N = 1840)*
Sociodemographic Characteristics of the Sample (N = 1840)*

Psychosocial morbidity

Of the 1840 children screened, 239 (13.0%) were positive on the PSC. Of these 239 children, 146 were classified as externalizers and 64 as internalizers.

Pediatrician recognition

The rate of physician recognition was low (6%). Physician clinical judgment in detecting psychosocial impairment showed a sensitivity of 20% and a specificity of 96% when compared with PSC classification.

Health care utilization and psychosocial morbidity

Results of linear regression analyses indicated a consistent relationship between child PSC-positive status and health care utilization. Socioeconomic and clinical variables entered in the model, with the exception of the child's age, did not predict additional health care utilization. According to the linear regression model, the average number of utilization visits was 4.3 in the year previous to the index visit. The model predicted that children classified as internalizers were 3.3 visits above the sample's average (P<.05). The model predicted 2 visits above the average for those classified as externalizers (P<.03). Chronically ill children averaged an additional 6.2 visits (P<.001). Overall, the model significantly established PSC-positive internalizing or externalizing indicators and chronic illness as predictors of increased health care visits (R2, 0.24).

When chronically ill children were removed from subsequent regression analyses, PSC-positive status remained a significant predictor of increased health care utilization. Within those regression analyses, children classified as internalizers averaged 3.8 more health care visits than others in the sample (P<.008), while those classified as externalizers averaged 1.9 additional visits (P<.006)(Table 3).

Table 3. 
Average Log Medical Care Costs for Children With Psychosocial Dysfunction and/or Chronic Illness and for the Total Sample for 1 Year Prior to Screening*
Average Log Medical Care Costs for Children With Psychosocial Dysfunction and/or Chronic Illness and for the Total Sample for 1 Year Prior to Screening*

Economic burden of pediatric psychosocial morbidity

Based on analogous regression models, the average log medical care cost within this sample for each child was $393. Chronically ill children showed a cost of $1138, resulting in a $745 excess above the predicted average for the year prior to the index visit (P<.001). The cost for children classified as internalizers was $412 above the average (P<.01), or $805 per year (Table 3). Because chronic illness contributed to a predictable influence on health care costs, additional regression analyses were run that excluded chronically ill children; the PSC internalizing indicator maintained significance, as in previous regression models.

Patterns of psychiatric spending, psychosocial morbidity, and age

Interpolated data from the same regression models showed that psychiatric costs increased significantly with the age of both internalizers and externalizers who were above the PSC-positive cutoff score. As shown in Figure 1 and Figure 2, the predicted medical costs of both indicators decreased progressively with age as psychiatric utilization costs increased.

Figure 1. 
Age effects on pediatric health care costs and psychiatric care costs among children with internalizing indicators, based on multiple regression models. The average log cost of medical care was $393 for the total sample for the 12 months prior to the index visit; for psychiatric care, $11. For the medical and psychiatric costs of internalizers, cost data were interpolated from the regression models.

Age effects on pediatric health care costs and psychiatric care costs among children with internalizing indicators, based on multiple regression models. The average log cost of medical care was $393 for the total sample for the 12 months prior to the index visit; for psychiatric care, $11. For the medical and psychiatric costs of internalizers, cost data were interpolated from the regression models.

Figure 2. 
Age effects on pediatric health care costs and psychiatric care costs among children with externalizing indicators, based on multiple regression models. The average log cost of medical care was $393 for the total sample for the 12 months prior to the index visit; for psychiatric care, $11. For the medical and psychiatric costs of externalizers, cost data were interpolated from the regression models.

Age effects on pediatric health care costs and psychiatric care costs among children with externalizing indicators, based on multiple regression models. The average log cost of medical care was $393 for the total sample for the 12 months prior to the index visit; for psychiatric care, $11. For the medical and psychiatric costs of externalizers, cost data were interpolated from the regression models.


Consistent with epidemiologic and community studies, 13.4% of children attending a pediatric visit were experiencing a clinical level of psychosocial dysfunction. Pediatricians identified 20% of the children who screened positive for psychosocial problems. Children with chronic illnesses and psychosocial problems had the highest levels of medical utilization. With all intervening variables included in the regression models, psychosocial morbidity raised health care costs and visits significantly. For age-matched children in the sample, internalizing symptoms were particularly costly to the health care system, with an average additional cost of $412 per year. Health care utilization was higher for younger children with psychosocial problems; however, the presence of symptoms among older children was associated with increased psychiatric utilization. The increased level of health care utilization of children experiencing internalizing symptoms has important clinical and empirical implications. In spite of the limitations of this cross-sectional study, its results have important implications for the understanding of clinical morbidity and economic outcomes in health care. For the first time, a standardized psychosocial screening procedure and a centralized clinical data-tracking system with comprehensive data on child health utilization, visits, and costs were used at an HMO to measure the economic burden of mental health morbidity within a pediatric population. This technology shows promise for clinical practice to (1) further pediatricians' professional mission of early detection; (2) aid in clinical decision making by providing clinical and economic outcome variables with which to better understand the service needs of patient populations; and (3) increase the appropriate use of health care resources.

A limitation of this study was the lack of external clinical validation of the parental-reported psychopathologic processes. The extent to which parental distress affects child functioning and parental perceptions of the dysfunction remains to be determined by future studies.38,39 However, the findings are consistent with those of studies based on representative national pediatric samples of psychosocial dysfunction.23,26 The clinical implications of the internalizing and externalizing indicators and the impact of comorbidity should be understood, and future analyses should address these issues in greater depth.

This study suggests the potential value of psychosocial screening, tracking functional status, and measuring health care costs for children and families. However, several questions are raised by the results of this analysis. While this study shows that increased health care costs are associated with psychosocial morbidity, the specific link between psychosocial dysfunction and increased health care utilization has not been explored. For example, once children's psychosocial problems are identified, it is not clear whether referral to mental health services would necessarily reduce subsequent pediatric health care utilization. These questions must be answered by outcomes research, which will determine future policy.40 Preliminary evidence in this study indicates that there may be a substantial cost that could potentially be offset by building higher-quality, more comprehensive mental health services for children. Further research is needed to prove this hypothesis by focusing on early identification and intervention.

Editor's Note: Considering the struggle our psychiatric colleagues have with managed care organizations, I hope the CEO (or whoever makes such decisions) pays attention to the results of this study. I can dream, can't I?—Catherine D. DeAngelis, MD

Accepted for publication August 16, 1999.

This research was supported by grant 950064 from the Innovation Program, Kaiser Permanente of Northern California, San Jose (Dr Bernal).

Corresponding author: Pilar Bernal, MD, Department of Psychiatry, Kaiser Permanente Medical Group, Suite 140, 175 Bernal Rd, San Jose, CA 95119.

Sharp  LPanell  RHMurphy  LOLewis  CL Psychosocial problems during child health supervision visits: eliciting, then what?.  Pediatrics. 1992;89619- 623Google Scholar
Institute of Medicine, Primary Care: America's Health in a New Era.  Washington, DC National Academy Press1996;
Brandenburg  NAFriedman  RMSilver  SE The epidemiology of childhood psychiatric disorders: prevalence findings from recent studies.  J Am Acad Child Adolesc Psychiatry. 1990;2976- 83Google ScholarCrossref
Costello  EJCostello  AJEdelbrock  C  et al.  Psychiatric disorders in pediatric primary care: prevalence and risk factors.  Arch Gen Psychiatry. 1988;451107- 1116Google ScholarCrossref
Costello  EJAngold  ABurns  BJ  et al.  The Great Smoky Mountains Study of Youth: goals, design, methods, and the prevalence of DSM-III-R disorders.  Arch Gen Psychiatry. 1996;531129- 1136Google ScholarCrossref
Lavigne  JVBinns  HJChristoffel  KK  et al. for the Pediatric Practice Research Group, Behavioral and emotional problems among preschool children in pediatric primary care: prevalence and pediatricians' recognition.  Pediatrics. 1993;91649- 655Google Scholar
Costello  EJ Primary care pediatrics and child psychopathology: a review of diagnostic, treatment, and referral practices.  Pediatrics. 1986;781044- 1051Google Scholar
Jellinek  MS Managed care: good news or bad news for children?  J Dev Behav Pediatr. 1994;15273- 274Google ScholarCrossref
Greenbaum  PEPrange  MEFriedman  RMSilver  SE Substance abuse prevalence comorbidity with other psychiatric disorders among adolescents with severe emotional disturbances.  J Am Acad Child Adolesc Psychiatry. 1991;30575- 583Google ScholarCrossref
Kazdin  AE Developmental psychopathology: current research, issues and directions.  Am Psychol. 1989;44180- 187Google ScholarCrossref
Forrest  CBSimpson  LClancy  C Child health services research: challenges and opportunities.  JAMA. 1997;2771787- 1793Google ScholarCrossref
Hankin  JOktay  JS Mental Disorder and Primary Medical Care: An Analytical Review of the Literature.  Washington, DC US Government Printing Office1979; National Institute of Mental Health Series D No. 5Google Scholar
Lozano  PFishman  PVon Korff  MHecht  J Health care utilization and cost among children with asthma who were enrolled in a health maintenance organization.  Pediatrics. 1997;99757- 764Google ScholarCrossref
Fosarelli  PDDeAngelis  CMelitis  ED Health services used by children enrolled in a hospital-based primary care clinic: a longitudinal perspective.  Pediatrics. 1987;79196- 202Google Scholar
Newacheck  PWStoddard  JJ Prevalence and impact of multiple childhood chronic illness.  J Pediatr. 1994;12440- 48Google ScholarCrossref
Kelleher  KStarfield  B Health care use by children receiving mental health services.  Pediatrics. 1990;85114- 118Google Scholar
Newacheck  PWHalfon  N The association between mother's and children's use of clinician services.  Med Care. 1986;2430- 38Google ScholarCrossref
Riley  AWFinney  JWMellitis  D  et al.  Determinants of children's health care use: an investigation of psychosocial factors.  Med Care. 1993;31767- 783Google ScholarCrossref
Horwitz  SMorgenstern  HBerkman  L The impact of social stressors and social networks on pediatric medical care use.  Med Care. 1985;23946- 959Google ScholarCrossref
Finney  JWRiley  AWCataldo  MF Psychology in primary health care: effects of brief targeted therapy on children's medical care utilization.  J Pediatr Psychol. 1991;16447- 461Google ScholarCrossref
Jacobson  AMGoldberg  IDBurns  BHHoeper  EWHankin  JRHewitt  K Diagnosed mental disorder in children and use of health services in four organized health care settings.  Am J Psychiatry. 1980;137559- 565Google Scholar
Starfield  Bvan den Berg  BJSteinwachs  DMKatz  HPHorn  SD Variations in utilization of health services by children.  Pediatrics. 1979;63633- 641Google Scholar
Murphy  JMKelleher  KPagano  MEJellinek  MSChilds  GMcInery  TK The family APGAR and psychosocial problems in children: a report from ASPN and PROS.  J Fam Pract. 1998;4654- 64Google Scholar
Tessler  RMechanic  D Factors effecting children's use of physician services in a prepaid group practice.  Med Care. 1978;1633- 46Google ScholarCrossref
Mechanic  D The influence of mothers in their children's health attitudes and behavior.  Pediatrics. 1964;33444- 453Google Scholar
Zuckerman  BMoore  KAGlei  D Association between child behavior problems and frequent physician visits.  Arch Pediatr Adolesc Med. 1996;150146- 153Google ScholarCrossref
Wakefield  JC The concept of mental disorder.  Am Psychol. 1992;47373- 388Google ScholarCrossref
Sturner  RA Parent questionnaires: basic office equipment?  J Dev Behav Pediatr. 1991;1251- 54Google Scholar
Jellinek  MSMurphy  JMLittle  MPagano  MEComer  DMKelleher  K Use of the Pediatric Symptom Checklist (PSC) to screen for psychosocial problems in pediatric primary care: a national feasibility study.  Arch Pediatr Adolesc Med. 1999;153254- 260Google ScholarCrossref
Jellinek  MLittle  MMurphy  JMPagano  M The Pediatric Symptom Checklist: support for a role in a managed care environment.  Arch Pediatr Adolesc Med. 1995;149740- 746Google ScholarCrossref
Unutzer  JPatrick  DLSimon  G  et al.  Depressive symptoms and the cost of health services in HMO patients aged 65 years and older: a 4-year prospective study.  JAMA. 1997;2771618- 1623Google ScholarCrossref
Jellinek  MSMurphy  JMBurns  BJ Brief psychosocial screening in outpatient pediatric practice.  J Pediatr. 1986;109371- 378Google ScholarCrossref
Little  MMurphy  JMJellinek  MSBishop  SJArnett  HL Screening 4- and 5-year old children for psychosocial dysfunction: a preliminary study with the Pediatric Symptom Checklist.  J Dev Behav Pediatr. 1994;15191- 197Google ScholarCrossref
Murphy  JMReede  JJellinek  MSBishop  SJ Screening for psychosocial dysfunction in inner-city children: further validation of the Pediatric Symptom Checklist.  J Am Acad Child Adolesc Psychiatry. 1992;311105- 1111Google ScholarCrossref
Murphy  JMIchinose  CHicks  RC  et al.  Utility of the Pediatric Symptom Checklist as a psychosocial screen to meet the federal Early and Periodic Screening, Diagnosis, and Treatment (EPSDT) standards: a pilot study.  J Pediatr. 1996;129864- 869Google ScholarCrossref
Starfield  BWeiner  JMumford  LSteinwachs  D Ambulatory care groups: a categorization of diagnoses for research and management.  Health Serv Res. 1991;2653- 74Google Scholar
Kleinbaum  DGKupper  LLMuller  KE Applied Regression Analysis and Other Multivariable Methods.  Boston, Mass PWS-Kent1987;
Canning  EHCanning  RDBoyce  WT Depressive symptoms and adaptive style in children with cancer.  J Am Acad Child Adolesc Psychiatry. 1992;311120- 1124Google ScholarCrossref
Canning  EHHanser  SBShad  KABoyce  WT Maternal distress and discrepancy in reports of psychopathology in chronically ill children.  Psychosomatics. 1993;34506- 511Google ScholarCrossref
Saravay  SM Psychiatric interventions in the medically ill.  Psychiatr Clin North Am. 1996;19467- 480Google ScholarCrossref