Demographic Differences Among US Department of Veterans Affairs Patients Referred for Genetic Consultation to a Centralized VA Telehealth Program, VA Medical Centers, or the Community

This cross-sectional study assesses care coordination and equity of genetic care delivered by centralized telehealth and traditional genetic care models for US Department of Veteran Affairs patients.

Genitourinary: Genitourinary refers to the urinary and genital organs. Nephrology: Nephrology is the branch of medicine concerned with the kidney. _ Personal utility: Ancestry: Ancestry is comprising of a line of descent or lineage; ancestors. Genealogy is commonly used to identify one's ancestry. Paternity: Paternity is the state of being someone's father. Paternal origin is relating to the origin or descent from a father/male. Transgender: Transgender is relating to, or being a person, whose gender identity differs from the sex the person had or was identified as having at birth. _ Pharmacogenetics: Pharmacogenetics: Clinical pharmacogenetics is the use of genetic data to guide drug therapy decisions. It involves variations in drug response due to genetic makeup and studies how a person's genetic makeup can affect their metabolism of a drug. Exposure: Exposures that might affect health, e.g., toxin, radiation, herbicide, pesticide, insecticide _ Pulmonary: Pulmonary: Any problem in the lungs that prevents the lungs from working properly. Affect the airways by causing a narrowing or blockage. Affect the structure of the lung tissue due to scarring or inflammation resulting in the lungs being unable to expand fully. Affect the pulmonary blood vessels. _ Reproductive: Reproductive: Reproductive health refers to the diseases, disorders and conditions that affect the functioning of the male and female reproductive systems during all stages of life. _Rheumatology Rheumatology: Immune-mediated disorders of the musculoskeletal system, soft tissues, blood vessels and includes autoimmune disorders _ Not Specified: A consult that does not fit any of the reasons described above, or no reason is specified in the note.

Method
To classify documents, we developed an iterative approach of classification, validation, and data augmentation to produce accurate results for Cancer, Gastroenterology, Cardiovascular disease, and Neurology classes.

Text Filtering
A critical component of document classification was eliminating irrelevant text in the document before any other processing is performed. Many documents contained standardized text that could be filtered out through manual creation of Regular Expressions and Filtering on a match. Filtering out information such as sending/receiving facilities, dates, form text (e.g., "Please enter the information below"), and other elements helped reduce the amount of text processed by other components of the algorithm.
Most importantly, this component of the software filters out lists of topic-relevant terms that are irrelevant for a particular patient or document. Text like "Enter patient family history of cancer:" is irrelevant to a particular consult reason but is common in documents and are removed during preprocessing. Additionally, terms with ambiguous meaning such as "CA" as an abbreviation for both "California" and "cancer" could be removed when an address was seen in text.

Pointwise Mutual Information
Pointwise Mutual Information (pmi) is a metric for measuring co-occurrence of two discrete random variables. The value of pmi(x; y) changes based on how much the observed probability of co-occurrence, p(x; y), differs from the expected probability of co-occurrence assuming x and y are statistically independent, where p(x; y) = p(x)p(y).

pmi(x; y) = ln p(x; y) / p(x)p(y)
When x and y are independent, then pmi(x; y) = 0. If there is a negative association between x and y then pmi(x; y) < 0. If there is a positive association between x and y then pmi(x; y) > 0. The larger these associations, the more the values deviate from 0. This can be applied to a classification task by calculating pmi values between words in a document and a classification for that document as a whole. For every combination of word w and class c, we calculate the pmi(w; c). For example, this creates a list of pmi values for the word "BRCA" and every potential classification ("CA", "GI", etc.) as well as the pmi value for every unique word and the "CA" class.

Stopword Curation
Stopwords in natural language processing are words that are filtered out before processing data. These are generally the most common words in the text or are words that do not contain meaning for the particular task. As part of development, a list of stopwords was manually curated as exclusions to the pmi calculations.
Due to the varied text in documents, there was a large amount of noise in the pmi results. Words that should not contain information relevant to the genetic consult reason often had pmi values that deviated far from 0. A few examples include "consult," "expected," "history," and "result." These words do not indicate a particular consult reason but may have surfaced disproportionately in one or more classifications by chance.

Document Processing and Classification
To produce output, each document was first filtered to remove any form data and irrelevant content. After filtering, the remaining text was broken down into a sequence of words. For each word in the sequence, the pmi scores were looked up and summed for each class with out-of-vocabulary words given a score of 0. The two classes with the highest scores were output with the associated document into the database tables. If all classes had a score of 0, then the "not specified" class is output.

Validation
To validate the system output, documents were selected at random from each predicted class and sent to annotators for expert classification.

Data Augmentation
After each round of validation, the annotated documents were added to the pool of training data and pmi scores were recalculated. The first round began using a list of the provisional diagnosis text manually classified into the correct class as initial training data. The primary goal of this was to improve the size of the vocabulary of the pmi values.
Because each class is so broad and contains many different mentions of diseases and genes (many of which would not have been seen in previous iterations), adding more training data decreased the total number of words in each document that must be given a score of 0.

Class Distribution
The following is the distribution of predicted labels by the NLP system. Primary classes are bolded. There are an additional 352 documents with the special category "NONE" that represent documents that contain no text after filtering out form text. These documents may be considered "Not Specified" but are not output by the system as such.

Precision
Precision, also known as Positive Predictive Value (PPV), represents the fraction of documents in a predicted class that are correctly classified. It is the ratio of true positive classifications to all predicted instances of a class. Precision for the "CA" class will be lower when documents are incorrectly labeled as "CA".

Recall
Recall, also known as Sensitivity, represents the fraction of documents correctly classified to all documents of a particular class. It is the ratio of true positive classifications to all instances of the class (which includes false negative classifications). Recall will be lower when the number of documents correctly classified "CA" is lower than the number of all "CA" documents.

F1 Score
Precision and recall alone have several drawbacks. Recall for "CA" is 1, a perfect score, when every document is classified as "CA" regardless of correctness, because all real \CA" documents are correct. Precision for "CA" is 1 when only a single document is classified as "CA" because all classifications are correct, even if most real "CA" documents are ignored.
To combat these drawbacks, a harmonic mean of precision and recall, called the F1 score, is also reported. F1 is a value that is close to the average between precision and recall when the precision and recall are similar but results in a low score when precision and recall are very different from one another. This encourages balancing over-and under-classification of documents for every class measured.

F1 = 2 x Precision x Recall / Precision + Recall eTable 2. Scores
The table below shows the precision, recall and F1 for the primary classes. All primary classes maintain an F1 score of around 0.9.