Conversations between doctors and patients can be wide-ranging and complex—unfortunately, a large percentage of the information discussed is immediately forgotten by patients.
Geisel School of Medicine researcher Paul Barr, PhD, MSc, believes these conversations should be recorded and shared to increase patient and family engagement in decision making and care management. For the past three years, he and the Open Recordings team have been working with Geisel medical students on a range of projects centered on developing and implementing scalable solutions for the use of recordings in clinical practice.
Because he lacks medical expertise, Barr, an assistant professor at The Dartmouth Institute for Health Policy and Clinical Practice, turns to medical students for guidance in the language of medicine—clinicians, of course, can and do provide guidance, but medical students, he says, are the perfect conduit between the medical and research communities. His easy collaboration with students stems from his commitment to mentoring students interested in research.
“Students have been central to the development, conduct, analysis, and write-up of a national survey of recording prevalence among physicians,” Barr says. “And they have also been hard at work on the development of the ORALS (Open Recording Automated Logging System) recording system platform that allows patients to record clinic visits with their doctors and uses machine-learning methods to automatically identify key elements of the clinic visit for patients to rehear at home.”
Machine learning uses complex statistical techniques enabling computers to learn and act like humans by learning from data that hasn’t been explicitly programmed.
To assess how well the system platform uses machine learning to identify key elements of each recorded visit—such as test results or medication discussions—James Finora ’21 and Kyra Bonasia ’20 review transcripts of clinic visits and tag segments for relevance.
“It can be unclear what doctors mean when discussing a treatment option, and as medical students we have an understanding of what’s important in that conversation,” Finora explains. “We tag relevant portions of the transcript about treatment options, for example, and also tag segments that are unimportant. Eventually, the computer learns what these things are.”
But trying to connect how people talk with how computers operate is tricky for a machine learning program—distinguishing between a disease and a vaccine isn’t simple. Finora says a doctor may suggest to a patient to get a tetanus shot—this is confusing for the computer because tetanus is both a vaccine and a disease.
The program needs to learn these distinctions in order to retrieve relevant conversations. When a patient wants to review what their doctor said about medications, the program will also retrieve a discussion about supplements, and how they may interact with prescription medication.
“In deciding what is important to teach the computer and how we are teaching it, Kyra and I created a guide—which is exciting for me because it is the foundation of the project. If the computer isn’t well trained it won’t function,” Finora says. “We want to teach the computer to adapt rather than change how doctors and patients interact.”
The medical students make such a difference—their work is critical to the system’s operation, Barr notes. “By working closely with myself, my colleague Dr. Saeed Hassanpour, who leads the development of the machine learning methods for the system, and the rest of our team, the medical students have refined an annotation guide that we are happy with.”
Finora says he has learned a lot about building machine learning programs in general—and “it has been really cool being involved in a project on such a foundational level.”
“One of the goals of the ORALS project is that family members can also access the conversation in order to help the patient. I think it will be impactful because there is a huge need for patients to remember what their doctor has said in order to adhere with plans, to be medication compliant, and to keep them from being readmitted to the hospital, which is happening with greater frequency,” Finora adds. “This can help patients—and that feels good.”
Barr says he often reminds researchers to be flexible and willing to change. “I always have to be that way because of the perspective medical students bring to the work—I’ve changed my strategy based on their feedback. They are doing very good research and I’m fortunate to be able to work with such a talented group of medical students.”
