Author Affiliations: Division of General Internal Medicine and Public Health (Drs Murff and Speroff) (firstname.lastname@example.org), Department of Biomedical Informatics (Dr FitzHenry), Vanderbilt University, Nashville, Tennessee.
In Reply: We agree with Dr Webb's concerns regarding false-positive and false-negative results associated with the use of natural language processing to identify postoperative complications within medical narrative. As we described in our study, developing search rules was an iterative process that required manual review of inaccurate alerts to determine how these algorithms might be improved. In some cases, there was a recurrent source of error amenable to a single correction. For example, false-positive alerts related to postoperative pneumonia often resulted from text describing the indication for a procedure, ie, “chest x-ray was ordered to rule out pneumonia.” Altering our queries to identify terms associated with diagnosis uncertainty, such as rule out, was a useful strategy to improve rule performance. In other cases, a false-positive alert was less amenable to a general solution. For example, a false-positive alert related to postoperative sepsis was determined on manual review to be related to the following phrase: “ . . . the patient is a retired septic tank repairman.” In this situation the search engine correctly identified the term septic, but we had not planned any exclusion rules related to occupation. Because of the complexity of human expression, it will always be difficult to anticipate all such scenarios. As Webb suggests, alternative approaches such as electronic/manual tiered approaches might be useful for rules with limited positive predictive values. These tiered approaches have previously been used to identify adverse drug events.1
Murff HJ, FitzHenry F, Speroff T. Natural Language Processing and Electronic Medical Records—Reply. JAMA. 2011;306(21):2325-2326. doi:10.1001/jama.2011.1781