Events Add an event Speakers Talks Collections
 
MLconf Online 2020
November 6, 2020, Online
MLconf Online 2020
Request Q&A
MLconf Online 2020
From the conference
MLconf Online 2020
Request Q&A
Video
Prediction Model for Favorable COVID-19 Patient Outcomes
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
48
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About the talk

AI in healthcare has significant promise for the quality of care delivered to our patients. In regards to Covid, multiple opportunities for leveraging machine learning ara available. In this work, we present an end to end development, integration, deployment, and use of a machine learning based model in hospitalized Covid-19 patients. We retrospectively built a model on 3,345 patients and prospectively validated on 474 patients to identify patients with favorable outcomes within 96 hours of a prediction, based on real-time lab values, vital signs, and oxygen support variables. In retrospective and prospective validation, the model achieves high average precision (88.6% 95% CI: [88.4–88.7] and 90.8% [90.8–90.8]) and discrimination (95.1% [95.1–95.2] and 86.8% [86.8–86.9]) respectively. We implemented and integrated the model into the EHR, achieving a positive predictive value of 93.3% with 41% sensitivity. Results suggest clinicians are adopting these scores into their clinical workflows. The talk will explore the technical and social factors with prioritization, integration, and adoption of a point of care model at our institution. We will also discuss relevant lessons learned and the special settings and workflows in healthcare that must be addressed for success.

About speaker

Yindalon Aphinyanaphongs
Physician Scientist at NYU Langone Medical Center

Physician Scientist in the Center for Healthcare Innovation and Delivery Science at NYU Langone Medical Center. Academically, he is an assistant professor and his lab focuses on novel applications of machine learning to clinical problems and research and infrastructure regarding model deployment and maintenance. He is also the Director of the Clinical Informatics Training program and designs curricula for training clinical data scientists to meet the needs of future employers. Operationally, he is the Director of Clinical Predictive Analytics. In this role, he leads a Predictive Analytics Unit composed of data scientists and engineers that build, evaluate, benchmark, and deploy (i.e. translate) predictive algorithms into the clinical enterprise.

View the profile
Share

Thank you all for joining us here in the machine learning and science session of a mocha. Today. I'm pleased to welcome yin and yang Fang. Director of clinical Predictive Analytics at NYU langone today, development integration deployment and use a baby validated real-time prediction model for favorable outcomes in hospitalized covid-19 patients and clearly covid going on. I did not decide. I just had not to show an image of that. But just to let you know right. New York with the one of the first

place is right to really see these bikes and Ben March and April, you know, our facilities are being overrun and we were getting you a I think a thousand admits the day. Some point today or something. Absolutely crazy hospital by the Stars. We have a lot of visits typical academic institution, a lot of positions, all kinds, of research Farm as well, and was very, very icy focused on. So a lot of the success of getting it has to do with the fact that you know, we are

IQ Focus been a lot of the connections. Can I see? And a hospital that leadership is one of the same time? So that makes it possible to play. My Beats X I run a prediction about a unit. So I am learning to text prosecution academically. I miss this professor a bunch of courses. And then also on the operations side, I run a team of a defense spending years. Our job is to provide ask you for a big project evaluation. We have a master's or Ph.D program a clinical fellowship and see what have a certificate program. So we're trying to clean organization that's used to pick him up and make me

sick and Based on data networks on Swan ballet Organization for 10 today, to science operations, with Microsoft, Facebook, and all that, all the head guys in terms of with their research arms out of doing Innovative restriction in healthcare. Do I am I here, I'm here. Really because of a broad frustration. So, Eric Topol publish this about six months ago and that bottom right-hand corner. You can see that, there's just so much work done on my medicine research and there's a lot that means there's literally, you know, just tons of

papers in early, in my career. I was one of those guys that was writing all these papers and I sort of said, you know, this is super frustrating because, you know, we're doing this really awesome stuff, but nothing making me come to practice and that was super frustrating. And so about 5 or 6 years ago, you know, we brought together. I've made a pitch to my leadership. Is he looking at, we need some sort of translational group that stole job is to think about how to translate that I bought at the point of care. And you're scoring systems especially are not no more new and medicine. If

anything they're probably the very first application of predictive modeling do things like porch has to Charles. Did you pass it to you may have heard of some of these they may not let you know as much as you think of it. Still are challenges try to figure out what those problems are. And why are there are some transitional barriers are some technical challenges. Do you know we really have poor coupling, a model predictions intervention? So, you know, I have a model that might predict readmission right now. So we're missing is, you know, you get discharged. If you come

back within 30 days. There's a penalty. This applies to that tradition, and it's considered to be the hospital's full mission. Right? Well, what am I supposed to do for that patient? And how do I ask, you prevent the transmission and sometimes there's a really poor coupling between why a person might come back and what is the inventions are off for friends of some hospitals, have difficulty compiling their data. A lot of hospitals are conglomerations of many different hospitals and might be running. All kinds of different electronic health-record systems. I'll bring it

at all together. Under One Roof can be very difficult. We don't have that problem and why you? Because we're all one system both and outpatient and inpatient World, limited evidence of what you applied at invention, does actually, you know, have an effect on your outcome. And there's typically you would do this in a randomized control trial, right? So you have a drug trial that was show on a drug with efficacious, and we just don't have a lot of that kind of data in the AI Healthcare space. It's difficult to climb out onto the point-of-care. So, you know, we have porch has Apache, the ones

they sell before they do exist, and sometimes right in the values, right to get up to get a consultation, but it makes it a little bit. It's even more friction and a recall. So there are bottles that that that work. Well, right with saying like the 25.9 days, but ultimately, write the predictive values are too poor to actually drive in intervention, right? And says, this is coffee on our mind. Whenever we know I present our models to leadership. Otherwise, the question is always well intervention

for patient. This is social barriers are terrible health professional training on how to use technology and when it does limit training at the leadership, level to understand bottle value workflow and aggression, and change management. There's one thing picture, taking bottles of that daddy wants it and cleaning kit. See what is they were trying to fix them outcome. And the bottle itself is just one component of this worry about other things like notification of excuse. So, you know, now that the model is

your produces a classification, how do I get that out to the conditions of the fighters of someone who cares? Write it and how effective is a communication Channel? What is, how can I drive that person to take action? And this is where I'll talk a little more about our covid-19 model and in the end of how we try, and what's the distance of providers reactions have been to the model and how we actually can drive an accident. Finally. What's the intervention? So, you know, I might have a readmission model, right? And I can tell someone that this patient. Mitted, they can take some

action but that accident, you know may not work. So examples of congestive heart failure patients have been for congestive heart failure you put on on Lasix or a diuretic. To try to get rid of some of the fluid but they're not accurate so that they end up back in the hospital right now. Though. You have an intervention right? You take an accent doesn't mean it's efficacious. By the time you finish, you have an outcome that you know, is that you have almost no signal because you have an amazing model, but you kept on losing signal along the way. She actually try to measure outcome of the

hearing. Of readmissions. So today I'm going to talk about this paper that we wrote about a real-time prediction model figure out from the house of covid-19 station. DJ digital medicine. You can find it. If you want. It looks like pretty quickly review of IT of trying to help providers are hostile. Make decisions about what to do. So we have models like you who's likely to have a positive test, right? So this was relevant, when we can have a lot of tests available who should be emitted from the stores and he was like these 62 shark in the hospital after the first three,

you know, you might say all those down like, amazing application, super useful, but it turns out, you know, they're actually not as useful as you think and nice but she didn't see anyone. For example. Yeah. Okay, great. You know, I know I can predict this. I wanted to rain in the next 12 hours. The promise of time was that there was nothing we could do for them. Right? We didn't have in their intervention that way. Possible. The leadership was like, well, you know, that sounds great. But not silly that useful. And the

reason is, you know, we have a lot of traditional methods of huis take to discharge. And what we found was that was some of these patients are some of the covid-19 patients, right? We didn't have to be as stringent as we would be with other condition that people might have it. We might be able to send them home earlier. So what does it mean for safe to discharge while he's a couple things? One is you don't get transferred to the ICU, right? That's clearly bad. You don't get intubated with his bad. You don't get come back to the ET so I can send you home. You don't come back to the EG.

Also, you know, you're really sick. You don't die. And you don't have options support and excessive nasal cannula is 6 liters. You know, I'm Health Services would not allow us to would not sensible option, it through this process. And, you know, you are trying to manage our bed flow, right? We sort of agreed to send patient, more oxygen than they used to it. So that's why, you know, we have the six liters per minute and the idea is to not predict spider, find a cohort who won't have these little. Can I augment his decision and trying to send a picture, how to do it earlier?

Do a couple different use cases of Provider. I want to know which my patients are at risk of Adverse Events so I can have charges so I can discharge the patient as a leader. I want to quickly know which faces the risk about prevent so I can follow up with the team as a care coordinator. I can quickly provide us with patient at lower risk of adverse charge. Are we develop our bottle? So, you know, we go to the model at all available and put then we did some feature selection to make it possible. He has built that Parks The Voice model within our are pretty

simple. We take a prediction time and then we try to see whether something happen. The next ninety-six hours. Our data was about two months worth of data about 3,000 location East prediction was a patient day. So there is 35621 days, patient days of which a certain number of positive and negative to try to predict whether that patient would be, would not have an adverse events in the next night. It's only on adult patients for over 18. Same definition as I said before, we took most recent lab

values. Benemax vitals oxygen rate, Austin devices day since admission, demographics, 60-20-20 split. We have actually felt an ensemble bottle and, you know, built one, very good model. And then we apply some novel feature, selection, algorithms using conditional renovation, test your basic rate, a simplified version of the full bottle, that utilizes the only 13 features. You can see here that the prep the brown is the black box full body and the parsimonious. It was the one of these speakers blue. These are the features

that are the model coefficients for the teacher. So after we get these double features, we apply a single logistic regression to turn these into explainable and just look at the beta's to see what the thing starts shooting at still classification are. We break it down as a red green and blue. And so, this was identified by our leadership. Also basing, our prior readmission, numbers about how many people we do readmit? And so this helps to drive the colors that we show the providers when they see this. See this interface. Critical role player, Grayson is

vitally important. So you need to tell people how this will look and what kind of date it will show you. And so this is how it eventually run. Still it runs every 30 minutes and dumb and it provides explanation. So here you can see a patient that's at high risk to the patient that has a low risk and you can see here at this time line right across what their trajectory has been overtime and the doctor said she would and you can see that the largest factor here is that, you know, there's not a very high resting heart rate there on induction device right now and they're bun is 39, and then

some minor cuts and then hear this patient is low, but you can see there are some factors that are contributing to the classification and but if this is able to look at these in first day of hell, yeah, okay, right. And then make a decision about whether these are legitimate concerns of whether there a weather station of Tactical. Would you have to monitoring? So we have at least two diagrams here? So this shows how often that display I show you before is refreshed. And this shows you how often people go to this Hopper bubble when they hover over

it. And so you can sort of see that as time is going on with John on May 18th, right? What are numbers have gone down as well. So there's just less less than less precious here. And this shows you that, there's a way that you can actually add this as a column in your display, which are showing a second. And basically, people wants to admit, they're calling Casey keep it on there,. They don't, they don't, they have it removed it. The model does perform a well overtime. So, when we go to model and may still actually has performance today, and that's important when you think about this issue,

is round mock draft. How do we evaluate this model? So, we may see, you have a digital randomized control trial. Here. You can see that the column I was talk to you. I am busy. All we do is we hide 50% of them. I automatically also the condition does not see the score and so are our goal primary outcome is to look at time from first prediction discharge our secondaries like this day in our in our second act as it is read, Misha's Wii versus a representation within 30 days, 30 days. So this trial is almost finished

finish the next 2 or 3 weeks. We were rolled a thousand patients or it's been a fight, a thousand patients. So we'll see whether this has a real stuff done on patients. There are some interesting observations which, you know, I just want to talk about because I think the relevant across other applications of AI and health care in AI in general and I'd love to get some feedback. So, you know, she has some faults one is you know is in this specific application. We have shown that it's significantly decreases provider and certainty about patient status and potential position. So I have it

at text Grant we're we're studying sort of the AI in the workplace and you know, we have these surveys has went out and so we've shown that this does help the positions of the provider's. Make better decisions are more certain decisions. About patient this position, a second is are Dachshunds overtime and I think this is a common concern with a are models. So in this case, you know, this doctor is pretty strong at first, but once the provider's metal model, has formed the provider's don't really use the tool anymore. And this is a problem, right? Because you know,

there are thinking is that there's fault Enos, right? No matter what mental model of the position, they come up with, you know, it's too complicated about of a task, you know, that a computer cuz you pretty well. And so, you know, we're not quite sure how to deal with this honestly, because at the end of the day, we're really concerned about making sure that surprising findings, right? Finding that may be contrary to what provider is thinking. Those are the one that we want to start this, right. Those are the ones we want. Solar someone that you know, what made us. And maybe you need to

rethink. You know, what, where you going to do for this patient? And so that's something that we've noticed is that, you know, people use a tool, right, they figure out a mental model, but once they have them at the model and just don't really use it for any more observation is not experts. And we, we've seen also that non-experts R12 actually benefit the most from this AI model because they don't really have a mental model or Around, you know, what, does a wallpaper look like it. So this gives them confidence, right above board and at the height of our covid, you know, Pete, we were

enlisting people that were not Australis, you know, in their day-to-day jobs to help manage the patient. In fact that we had certainly quite relevant. Unfortunately, or fortunately, you know, our number for benlo. So which do, you know, everyone is take care of our covid patients. So bottle sharing this model is available and easily shareable. We have the partner institutions who got this bottle running in about half a day and we made it with our vendor. So that is easily shareable. So if you represent in this institution, and this is of interest to you, you'll please

let me know and I'll just take a few seconds and talk about success principles of someone is workflow first. So, you know, we thought about what the problem was first and then we built the AI model and using how scared never works the other way around, you know, where the bottle first and daddy, try to work flow. We had a good leadership by in all the way up to our being on. This is super important because there's a huge team to have to come together and make this work and that was only possible, but their support a false positive, it will cost, right? So like

someone who shouldn't go home, like that's a real cost. And so we're always very cognizant. Possibility and try to mitigate it at all times. We have clear organizational boundaries about who's responsible for whatever organization of the safety of migration in front of model building. And that helps a really, really nice who has responsibility in either going to do what I can and that's really important. And finally, for us. This is just a data point, right? That I provide to someone to say. Hey, you're just another something right to either

support your decision or not the Fortress decision. If you make a decision about how to integrate that to the rest of the plan is going on. And then finally communicate the surprise this what I talked about earlier. No, we haven't really come up with good ways of communicating. You know, that a patient who you might want to discharge, right is actually not doing well or vice versa. This patient is not doing well, even though you think they're doing, okay, and are actually anyways. And so how do we communicate? That's the price, right? So now the people aren't going to 20 more to look at it.

How can I communicate with them on when their decision making? Is that correct? And then they can be brought the tool can be dropped in them to show them here. Some information about someone that disagrees with you and be very similar to, you know, in our Hospital teams, right? The timing may say, you don't like to think that takes is ready for discharge as a resident, or in a warehouse that make it. Well, you know, I don't think, you know, we should do that because of XYZ at interchange, right? The human nose that is trying to make this decision. The

computer to know that there's a decision that the pending and that what is suggesting is contrary to that decision so that it can be brought to the Forefront. This is hey, this is that that they may not be, you know, what we're working on is how to communicate here to make this work, cuz it's Jens leadership might get the science team. Thank you. I'm happy to talk a little bit after the session. If you want. We don't get the benefit of being a meeting each other. After the beep. I could talk to me more,

but please got this number down. If you can afford is Noom and I'll be on a little bit of a specifically. Thank you, any questions. Hello, can everyone hear me? Do have some questions from the crowd group? Carlos Alfonso would like to ask. Could you please explain a bit more about how you did decide how you decided the red green? Right. So what we did it, we look at our readmission rates and that's how we set the green threshold. Because we said, let you know,

whatever your build has to be at least, you know, not worse, than our current rates. We looked at our current rates and then we have to leadership. We just said, hey guys, you know, what, what it, how often is? This model predicted to win? It's okay to go home. How often do you want to be right? And said, you know, based on our race who wanted to be about, you know, 99% So that's how we chose to value and in the orange and red. We just could have chose half of it. Because at the end of the day, you know, people are going home, read for sure. And that warns, you know, it's not really that mean.

We we could have just made it to Colors, green and red and just called it in a call to the day. But it was really mean that we are focused on and making sure that that prediction was correct. Expression. Can you guys hear me? I cannot hear anything anymore. I was on mute apologies for evaluating the model. How are you measuring that the providers? By the Square open fuse? How does his overall trend affect the results of the digital are CD? One of the huge worries, right? That we have an end. If there's a reason why we might have

a negative outcome. It might be because of that. So, you know the tools we have in our EHR for monitoring who like well, yeah rollover cover bubble for him or actually quite limited and we can't get another one. We don't know. These are some recent istics. Right? But we don't actually know what providers actually use a tool and what do not a future version. That's a feature that they promised us. So that way, you know, I can tell. You know, we educated like all the hospital as but only 50% of them are adopting it, right? That we can go out for

a few know if the other 50% is that how you know, you really need to use this fool and in drug Carlos. This is very similar to adhere. It's right for drug medication. So you might run on RC key for a drug and usually their research assistance, right. And still job is to make sure the patient's take the drugs throughout the entire time. They're on that way you could actually show that the truck is efficacious or notification and try to do that. But you know because of the large scale of this, you know, we just didn't have a way to communicate with everyone every day. Does. Hey, are you using

this tool? You know, what are you doing with it? And so all of our analysis is a little bit. To in terms of, you know, just hoping that before using it and hoping that people are seeing it and we'll see what happens. My connection is that Darcy is going to be negative and that we're not going to see that half-day mediately just a decrease or we're hoping for but not because it wasn't useful yet. But because your people are necessary sing it especially the experts that has talked about earlier. That you see, you know, that did see the model early on like once

their mental model has formed in what state is corroborated sort of like, oh, yeah greens are definitely. Okay. They try to stop using that the model because, you know, they don't know when to go see it or when they're going to get something different, but we'll see. I mean that it is there on the people of the dashboards as I saw this picture to your right is that column? So, you know, maybe it'll, you know, catch their eye and they're thinking of discharging someone into this red, you know, then we'll know that then don't go in and look a little deeper to see whether it's legitimate or not.

You for that answer, and thank you for your wonderful talk. Thanks everyone. I'll be jumping over to the zoo. So if you guys are interested in joining me, just showed up there for a few more seconds. Please stop over. And I'm glad to talk to you in more detail about anything for her. Yes, and I think we have to move on to the next speaker, but there were some follow-up questions. So I'll Carlos is your question. If you have time, please join that and soon. Thank you.

Cackle comments for the website

Buy this talk

Access to the talk “Prediction Model for Favorable COVID-19 Patient Outcomes”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “MLconf Online 2020”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “Artificial Intelligence and Machine Learning”?

You might be interested in videos from this event

February 4 - 5, 2021
Online
26
104
ai, application, bot, chatbot, conversation, data, design, healthcare, ml

Similar talks

Galina Grunin
Distinguished Engineer at Optum
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Josh Wills
Developer Without Affiliation
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Patrick Hagerty
Chief Data Scientist at Arena
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video
Access to the talk “Prediction Model for Favorable COVID-19 Patient Outcomes”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
949 conferences
37757 speakers
14408 hours of content