About the talk
This video is part of the R/Medicine 2020 Virtual Conference.
Play for life. So welcome everybody to the last talked of this block. My pleasure to welcome dr. Anton Becker. He's going to give a talk on building, a radiology workflow manager with r. Thanks for the introduction to everyone and welcome have already seen a great some great this morning. So happy to Idaho Falls. I'll deliver. First of all, a big thank you to the organizers of the sponsors, fantastic events so far. I also need to acknowledge my colleagues and friends from the Kettering Radiology. IT band stretching the patient seems especially those wonderful
people who have really massively help getting this project off the ground. I'd also like to thank our Radiology faculty and, and our service Chiefs and our department chair. so, Couple of words about myself. I'm a radiologist at Memorial Sloan-Kettering since last year. So I'm an MDM coming from the, from the MD sides, and just the slides, or just give you an overview, I did learn are in the, in the context of my research and scientific work. So I'm not a software engineer and I'm not a computer science
person, but yet this program actually has to have some kind of more to do with with engineering at a lot less with this time. So since I guess there are going to be only very few Radiologists on here, I want to take a few minutes to explain the radiological workflow. This picture should you don't need to study everything that's like you're in detail. I just want you to take a few and a couple of stick white buns from this. So, the first thing is, it's not as simple as you might think to just go from a medical image to like the results. There's a lot of intermediary
steps involved, that's the first thing. Second point is, if you look at the at the far-right Circle here, you can see that this is that this is a some reading the chat right now. So I can back and maybe interact with you guys this. So we have two tall is just on that. Call very good. so, Everything on the right side. So this is kind of the Legacy workflow actual films that were taken and that were them developed. So this in the, in most western Radiology, departments, that work flow doesn't really exist
anymore, but there used to be a physical room where people have to go and take the film's off the physical folder and then go with him to the light box and read them. So as you can see, kind of hear the digital workflow that we have is kind of modeled after this old after this old workflow. It doesn't on a technical level. There is no reason that needs to be like that, but I think this was just done so that all the radiologist to where used to be old workflow, they wouldn't have to completely change their workflow. It was, they could just be told
look like whatever you did before physically, that's now done virtually on a computer. So first take away point, the Radiology workflow, as we know it today is kind of a legacy System model after an old film workflow. And the advantage that this gives us for this project, is that usually those little parts that you see there, they were introduced one after another so often times there, a different software is involved and they all need to talk to each other. So there's a wealth of the ice that we can leverage some use HTTP. Other use Healthcare specific
standards, like the hl7 standard or the back from Center. so, This is an old work slow. This was kind of old design for the for the Old City Radiology system. We as Radiology has evolved as a field and we have a lot of new problems. In our opinion, those new problems they need new Solutions. So one of the new problems is that the size of Institutions has increased constantly about 15 or 20 years ago Memorial Sloan-Kettering have maybe 10 15, radiologist. Now it's a hundred and sixty
real. Just so like like a gourmet restaurant if you want it, if you want to deliver high-quality work, you can just use the same tools and scale things up in like, indefinitely. You'll never find a three Michelin star restaurants with like 800 tables. Because of that scale, you won't be able to deliver that kind of, that kind of quality anymore. So I'm in our workflow, this would mean either you'd have huge cluttered worthless so he was worthless where you have to look. For exams. Or you have a lot of small different predefined work list with
like certain criteria of people would go and need to click all of them. And so it's, I saw it all. Just introduces friction. If the workflow doesn't make it run smoothly. Our solution to this is that we use our to send the eliminations directly to a person like a few of everywhere, every radiologist. Second issue is sub specialization to wear as a back in the day you have one radiologist that you had a brain scan. She would read it. You had a scan of the lungs he or she would read if you break your leg York. The same
radiologist just read it. Today we have a very specialized Radiology as other fields actually have moved into the more sub specialization. For example at Memorial, we have a team of radiologist that deal exclusively with Cancers of the kidney. Bladder and prostate, are we have another team that deals exclusively with chest, kind of diseases of the chest, lung cancer, Excedrin, That's one that fit that the last point is and where it's back in the day. It maybe was acceptable to wager a few days for the results of your scan nowadays. People usually want
the results quickly. Like, I mean, it's become very apparent in the covid crisis. You, you do a covid test and the results in a few hours. At least you don't to wait 2 hours for your test results. Similar in Cancer Care where you have a Cancer and you're worried that that's it. Make it worse. Or if my Rick her and you don't know. If you actually have to stand, you do want to wait a week for the for the result. You want to have a quickly. So we view this as bad as a optimization problem. Basically, we optimize between the available Radiologists and one side and the unread examinations on the
other side and V. We basically Optimist over this with a customer at the Carlyle Group which we implemented an hour. So if you think about this conceptually I basically about the surface, you see down here, this is the difference between the two distributions. So red means fewer than the rest by the by the appropriate. Stop. Specialized radiologist and blue means a lot of the area, a lot of thistle specialized exams are going to be right by the specialized. They all just. So, basically the way you do, it is basically throw a whole bunch of
marbles in here and you look with marble and some the lowest point and that's your, that's your ideal point. So short schematic flow chart about how the program that she set up in the in this box to see ignite, which is the the project name for Eye Works a manager. We run it actually on an art studio connect server. I'll talk a little bit more about that in a minute. We do mirror most of our clinical systems in a SQL Server just so we don't have to interact with all of those. It's not strictly necessary, but we do it this way to agenda is our agenda
is our system for making be clinical roster, or scheduling which real just work on those days. We all some mirrors at this Center in a SQL Server. So, basically, that the basic steps that that ignite does is it takes all of the unread examinations from the sequel server. Its first of all looks which ones of those are obviously still open, then it will assign us so-called DMT week of his disease management team. So this is basically, which sub special session exam. And we been assigned to different algorithms. I just saw a
question lives and in the end, I think it's a great father. But some people don't seem to like it. So if it's not a gradient descent algorithm. Actually, if you think about it, in terms of, in in machine learning, you have a huge search space. So you need to. So, you need to use me to very sophisticated algorithms, to find that minimum point, we just use bootstrap and we just do random permutations. And that's, that's enough. You think about it? It's like the train, a neural network or something. You're trying to find the minimum
points in landscape the size of the state of New York. So you can going to need a map for that. Where is the size of the search space? In an hour cases? May be the size of this table that you're sitting at right now. So so you won't need it a supposed to get it, I'll go ahead and send that you can just do it randomly. So our system has a couple of of neat things implemented. So that this is an HTTP API, very simple. And this is how we assign the exams to the different radiologist since their personalized, cute. We do have a couple of neat
system. One of them is ignite will look up the next patient appointment. So if a patient has an appointment within the next 24 hours, with the, with the clinician met with her oncologist, then the radiologist was assigned this exam, they will get an email. Notifying them of course, is especially relevant examples within the next hour or the next 2 hours. Then the radiologist can prioritize the results available before the patient, has the appointment with her Clinic. That's basically for that, a couple of details about the RMS limitation.
We've decided to write this as a package because then it's portable its insurers. A certain level of consistency because then we document every function. And we also have some sort of integrated quality control at first of all, when we built the package, second of all. We also used at Travis and tests that to kind of ensure that everything works like, it's supposed to work. Our Comm code B group it into four different functions which you'll see in a minute. When I, when I show you an example, we use core functions which have the optimization algorithms implemented,
and then we use a bunch of functions that just interface with those. I'm single and our interfaces, we have certain auxiliary functions, Which business mostly for a logging or a, for the use of the pens package, and we have a couple of internal functions of the smaller functions, that kind of make the code more readable or more elegant. The second important part is in trying to look up tables. So you, you obviously need some sort of a priori information in your in your in your program. Like for example,
examine the program is she allowed to sign to to examination at 2 to the radiologist for ship. It also needs to know which radiologist has which expertise. So I all of this is what I've done with it. With a bunch of stuff. They took a bunch of CSV files and some of them are all supposed to wreck lee from internal databases. Let me have the so-called module and that's just a fancy name for rmd documents, which is a run at even intervals on the art studio connect server. A couple of words about our studio connect versus a standalone server or some other self-made solution.
I'm a big fan of our studio connect. We use it at Memorial for various analytics and dashboards. So it was just, it was available before we started this project. The nice thing is, it's, it's has a built-in pin board for use with the pins package. It's integrated seamlessly with rstudio. So, whenever you're off that your package, we just have a separate update script that then just from our studio updates, all our modules automatically. So it's those very, very painless. The whole process is nice, high-level
functionality, just kind of generate reports and email different people. The downside of what, of course, it's a proprietary product. There is a cost involved which you might be able to achieve at a lower cost. If you have like a Linux server with only running open source software, however that might that might, on the other hand, the sort of that might be Might be. I lost a friend. Yeah I might be more intensive time-intensive to to set it all up into my playing at. So my my suggestion would just be too to
use whatever you have available and then run with that. So can we have an example of our remarks and document? This is like a minimal example. The true module is a little bit more elaborate but basically first you first, you loads the library ignite and then I looked at the market or for the pipes and things packaged and a kind of logging. The first song Shania, whenever whenever I don't use the package name, it'll be an internal ignite function, send the first function, reset the HTTP token for
the whole session. Next function, we were just an hour inboard on our studio connect, In the next, all of those kind of things that you can see the very pure, our way, the next line fetches, all the completed examinations, and then the signs, the assigns, the DMT to them. In the next line, I get all the radiologist to say on a CT shifts on that morning. I'm closer to our native dataframes and I then this is one of the core functions that I function. This then divides, everything else between the
radiologist and here, I can actually Define the number of bootstrap were a number of directions. I use 50 in this case. And then it'll generate a list of three days of friends, actually, CTS, and all the cities that were actually successfully. A signs are going to be going to be on separate. They hit a frame and this Talking Ben actually communicates with the dictation system and does the actual assignment. Every, every event has been assigned, as I get to get Survival, that gets a sign status is. Okay. So I still throw those out and then I'll
send them to my pin board and after that, I do different stuff with them. For example, in our case, after another R&D actually takes up that pin and then looks at which patients hasn't have a 7 examination coming up and then does the email notification. Some results. The code base. The initial prototype. We built over a weekend or over a over a week, had $18,000 and fifty lines of code. So this is just a show. If you want to try something like this at your department, or if you want to try whether it's feasible, whether it's possible,
you can do a very fast prototype with r and you can very quickly see whether it's possible or not. The reason I actively developed this program for more than three months in 66 Custom functions. And still the code base is still quite slim. It's still less than 2,500 lines of code. Looking at the complexity of the different code, some of you might be familiar with the cyclomatic complexity. It's a measure of how complex a function is and you can see all the Divide functions. Those are all those core functions with the implementation in them, those are among the most
complex functions. So what is the actual effect on our clinical workflow? You can see how are over a 2018-2019. Those are just the months of June and July 2018-2019. You can see that the number of exams has steadily increased every year. This follow the trend that was already present in the prior years. And what has especially increases the number of exams that were Red beverage slower. So more than more than a day ago and you can just see that well, that the number of exams has has still increase. We've managed to read becomes a lot more efficiently in 2020
despite every reader, not reading more exams. So we so we were able to keep the number of exams respirator. Those were actually stable. So we didn't force people to read more. It was just it was just distributed more efficiently. What does has also enabled us to do is actually beforehand when we did not put system. The first thing that would be kind of called when there was sort of an overhead of examinations available, the first thing that would become was admin researched. This you're actually able to give people a lot more of that Edmund research time which of course results
in a lot of pain. A lot of happy colleagues don't have positive note. I also need to mention some of the challenges we have. Of course if you limit the new workflow at first, there's going to be some resistance to any change that you introduced, you do need a minimum number of Radiologists for day. Of course, that's a planning issue. Otherwise, the best algorithms not going, it's not going to help you. I'm the premise of our implementation of this was a balance clinical productivity, which also meant that I'm some readers. Who are a bit slower. Are there, disproportionately
affected, right? So they're they're going to be assigned more examinations then then faster readers. And from the technical side, documentation testing, testing testing. And of course that goes into ensuring continued support since this is a in-house application. So conclusion and closing remarks, this is the first Radiology workflow manager written in our. We think it's the start of a new personalized Radiology, workflow Paradigm, and I don't know. That's just the first pure. Our applications is actually used in clinical care, but maybe I'll find
out something differently. And in the process of this of this event and our helps us to deliver the best possible care to our cancer patients. I was that I thank you all for your attention so I don't have to question. This is this is a fantastic manager so why did you choose are for this? Uber perhaps more traditional language is like Jonathan cetera. So that the main, the disco's this goes back to the sort of what, what kind of problem are you dealing with? It's an optimization problem and you're dealing with, with, with data on, with data frames and mess
that they are friends and it's just as a Shona, I guess 2000-2500 lines of code. So, it's, it's fairly simple code in arm and in, in in our pain, in the arse, just like the best tool for the job here. Surely the same thing could have could have been implemented in Java python the receipt, but I think the whole given that our sister powerful is the ideal have the Ideal tool for the job. The other time I can dry can actually read it. So the package is not open source so far. Any thoughts about making the open-source? Yeah, I thought
about it right now. I have a lot of Memorial Sloan-Kettering specific thing still in there, but I'm definitely see if there was any interest. Of course, it's a lot of work making it making it open. So it's nicknamed flickable to everyone. If there is any interest actually consider making a big fan of Open Source software, And challenges getting management support. Well actually it's so this is actually I was kind of like I was working with this project started with a lot of tail blink because our is already pretty
strong and our departments. So so that was fortunately not an issue, we have to deal with you. I said how many lies of tests that are in this car actually. Not as many as it should. Thank you, wonderful talk. So, we have a break right now about 8 or 9 minutes, and then we'll see you in the next session for our last block of talk for the day.
Buy this talk
Interested in topic “Medicine, Health and MedTech”?
You might be interested in videos from this event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.