Dan is the CEO of Decision AI. Decision AI helps data scientists deliver better business results from their existing machine learning models. Before founding Decision AI, Dan worked for Google where he created the world's second largest AI education platform. He has contributed to the Keras and Tensorflow libraries for deep learning, finished 2nd (out of 1353 teams) in the $3million Heritage Health Prize data mining competition, and he has supervised data science consulting projects for 6 companies in the Fortune 100. Dan has a PhD in Econometrics.View the profile
About the talk
Machine learning is a technique for making predictions, but businesses are broadly struggling to turn these predictions into better big-picture business results. The problem is that data scientists typically do little to rigorously the business rules that they apply to their machine learning processes. This talk shows techniques to use machine learning outputs more efficiently and deliver more bottom-line value from the same predictions.
Tend to how to translate a I could feel decent A Better Business results before that. I've been doing machine learning and AI for about 10 years during that time. I got second place in the machine on her competition over 33 years as a defense Google Adventure companies in the Fortune 100 and I'm going to start with an overview what I've seen in our field so really you only even if plastic sealer that we've gone through three different stages. But there was a. Of time when data Fife organizations were
allowed to just do instrumentation. They didn't need the results actually feed into real-world business process and have you seen this is really captured and sometimes by well-known projects alphago is one of the most famous AI projects the last few years open a I had a Teamster millions of dollars on computer take some of the best resources in the world. And what are they called with that? They built software that could do a better job in the best units it playing video games and for a long time businesses
have their imagination captured by the identity models for Grace smarter than a I was very smart and they tolerated even like to spend money on dating sites in the i-team's Shirley for experimentation to keep the Are you a couple years ago two or three years ago that really shipped it in that I think now that most AI EI teens and dating sites organizations. They are required to at least get them bottles into deployment that's become much easier recently, especially thanks to some of these logos on
the Lesser or the tools available through all of the two major public land so gcp Microsoft easier than ever to take bottles and get them deployed even companies that are not the least of the clouds have a lot of great tools to make MLS season ever and as a result we see the number of machine many miles to Floyd is really skyrocketed in the last year or two. But the problem that we are facing as one more of these pallets at the Floyd is that they make predictions for pieces that are a very narrow piece Mark Stokes a friend of mine works in in
Defiance for company that for real estate Vape machine learning models and they forecast for any given price that we said on all our future Hotel. How many rooms will we sell only one night to sell hotel rooms and he could figure out how is 10 nights at a hundred $50 or five nights at least our price of turn it on it then that would be which of those going to be more money. But in a world where you got things that play out of a longer. Of time, so pretty since we've got rooms and whatever we don't sell tonight. We can sell
the next day whenever we go to the next day after that now these predictions for what will happen in the next 24 hours then actually in sufficient to ensure we make good big picture positions that optimize things like Lacroix. I found people who are not dating site just surprised to realize how limiting conventional supervised machine on again and how we're not able to optimize for the dynamic Frosty's Frosty's in the world play out of the series of steps to dating site scams are increasingly finding the deployment is not
enough and they now need to go beyond experimentation where they were a few years ago beyond the point, which is really where we where we been recently to ensuring that their mothers are useful. Switch Eve, not just narrow goals, but they're advancing a large picture Business course of action, which everyone is doing and then using those predictions we should get out of commission do optimization and ensure that we get practically I just choked. I'm going to walk through the process to do that in a pretty complete way for one particular example.
and this is an example of the Galaxy quite widespread in Massachusetts in part because of what the Simplicity and the fact that this is a use case that is is very very widespread from here will apply for promotions cases for my machine on a disease of building a model to handle shrimp with an insurance policy. They get services for some time and the business wants to figure out who is likely to charger. Common way that we show the output of this bottle is used to measure the answer people were or not. These tax
percentage years that we have here two different rows. So we look at our model we safe for people who actually did term that do the top row how many of them are predicted tomorrow to turn in this case. I could be 50,000 for people who were who were who actually did shirt how many they were crazy enough to get 10,000 on the bottom row that people who actually did not certain. How many did this is a really common output from a machine learning model where that they decide just what they here's a summary of how good our results are. this is
really despite being a widespread visualization of our results this is not very actionable yet. The first step to think about this in a way where we are least Stern to move towards be more actionable yes take the same results I've still alive I still kept the rose mean the same thing so the top row with people who actually turn the bottom row is people who didn't charm but in a columns and replace the word Prediction. When do we take action? The key is if we make predictions about who's going to turn our predictions don't methadone affect anything. What
matters is what we do in the case of people who got shot. So a common example of emotion that everyone who we thought was going to try and encourage them to stay with our service or we might call them in many cases do even offer promotion and get the discount. If they ever thought they were trying to give them a discount if they stay or if they renew their service. Here we've taken it from where we predict to if we take those predictions and take actions on
them fall into each of these buckets. Lexington this weekend. We can get more granular about what we mean by Casey actor because it's quite common or people assume that the probability. So if we score everyone of what is the likelihood of that person turning when we say we're going to take action only on people who is likely to return is about some threshold. We can move that track on all the way to the left and see where it takes action run a promotion to everyone.
We could move it all the way to the right and say we're only going to take people action and horror on the highest. Kefir be most likely to turn in some of that sense that we had on that table of who do we take action on That's something that we now have a lot of control over because we can take the boys to the friction for that model and determine what the criteria to determine which people we take action on. All right, so we still are are not yet optimizing spritzer bottle lineup. So how do we do that given the fact that we can
take the predictions from that model and do something. So it's a free stuff in a moment. I can talk about things are more sophisticated than this but this is simply to say we evaluate for each individual individual to actually we're going to turn what do you think the value for a business's of taking that person there predicted to turn and now running a promotion in this example a $10 and you can figure out what is the value of a given sum as actual outcome which UK Yeah, he's got a shortcut across
Foundation to figure out who wasn't going to charge. We if we run a promotion for them take action and we do it on necessarily. What is the cost of that table in the bottom is a cost-benefit table. You don't need the column on the right to be zeros you could assign that other values in practice. The only thing is going to matter forward for the decisions that will show you how to make tempura batter don't even matters at the difference between First Column the yes column in the know, that
we can evaluate. What is the business benefit of running a promotion based on the foot of our machine learning model where we used a given trait that threshold determines how many people we run a promotion to and as a result determines all the values in this top table. Are we can simply do some some new era calculation to say? Yes, we use that for soul in this case will run a promotion to 50,000 people on average of a valuation of $10 per person for for running at promotion attacked off the 30,000 people who were unnecessarily rear end up wishing for maybe a man. Are we
and that means that in this case, we got a valuation from running this promotion when I look at all the people who is correctly Ranch to as well as those we meet us here anna to we can see that the value of that. We're still not we now have a way of measuring the value of using a machine learning model to Target a population and then run a promotion based on that, but we're not get off the bus. Shut the way that we ensure we are getting the best business outcomes possible. Given the
spring work that way down just to now for every given freckled say what if I run a promotion to everyone is at least 1% like these for each of those thresholds. It was a sinner thresholds. I will get different values in this table the topic of changing the population of of who I actually somebody has the promotion to every time I change these values on the top. I can just do some numerical analysis and figure out what is the what's the benefit of running a promotion to. Population and now
I can trace out. What is the value to my business of running a promotion given any decision special? So what are we done so far? The mathematics of this is actually pretty simple coding python regularly in minutes, but we've done something which is far back, We are gone from I've got a machine learning model and make predictions. What do we do with us Productions typically people sit around and talk about what to do or the data science with something which seems reasonable but it really is a strategic decision
and now we can rigorously optimized for what is the ideal decision-making rule that we use as a weapon or not? Sure. So, like I said, it's pretty simple and what is really important limitations the most important limitation and it we can talk through threw in a bunch of limitation is that this machine learning model has such a small fraction of a large picture business process Indigent people who don't work a lot with data scientist or really as they become closer to understanding that you can build a machine learning
surprised by this but machine learning predict and very specific acts as a result we can have this and it's just a manufacturing so then we can have the manufacturing starting with all sorts of relationships that play on over extended. Of time. Anything that would do with a machine learning model. It's a predict. Something is very narrow and very specific statement about the way that didn't they predicting that need to be very narrow. We are like today to change to a world where we can do what is over here on the right where instead of just predicting one narrow piece
and then coming up with sore back guess about how to use the predictions for that in order to optimize for a big picture business. Kpi. We'd like to build a model with everything that matters from The Sting that we control all the way down to our business of okay. I think it is very important specify a business public API because we want to optimize model-actress what we wanted for the model where you can say everything that matters is all at the exit for is someone that is certainly not the case in if we go back to that cost benefit table
there. We just really had a best. Guess what's the value of running a promotion today individual and if it wasn't really calculated by To pursue this broader vision of of what it means to build a model. You need to integrate multiple machine learning models because they are basically just collect them and Manufacturing. It is time to find someone in marketing and then bring all these pieces together into one holistic model so that now I can say given this ballistic model I change labor if I
increase at a certain amount or decrease a certain amount. What is the impact it has on production and was in fact that hasn't failed as a revenue if there's some awesome except what I want to do. All of that is my bottle so that now I can buy a difference. The centers for my labor and actually see the bottom line here comes to an outcome of like a holistic profit in a way where these models in or no longer silent. What is to give you an example of what that looks like?
It is trying to get what we talked about. That could be for every customer. I say I can put in a given amount of promotional effort that's going to influence for that to a customer. Does that customer leave or not? So we could have a machine learning model that could influence the revenue if I cut them a discount I could get the formula for what's the revenue did I make with that commercial? What is the record without it made? It is a fixed cost of providing services in each
of these arrows. Does it cost to leave estimate with a machine on Apollo? These other areas could be could be summarized in just the creation and now I have a system of equations where I can I can figure out for each customer. Do I make more profit if I come up to them ordering a for-profit if I don't and we now have a specific? Individualized personalized what is the ideal thing to do for this customer in order to whatever goal is expected profit the
profit. So we could have a single customer here and we have to program it says a model and it says what happens if we offer this person or promotional discounts what happens if we don't the First Column is what's the probability of that person staying with our service? So that's something for the machine on your model profit if they stay that could be $100 and now is it smells location are expected profit a month money if they say is $90 do the same thing for the subsequent month. You can add it up and we can see if we on for this person expected
value. Where did Ariana side say? What if what if we run this promotion and we don't offer us at what if we don't offer that person the promotional discount now, we make more money. So you see profit if they stay because you're not offering a discount at the higher value the probability of them staying because we're not offering a discount in each month is slower so far. It's it's 80% * 90% in value could come from and if you could add that I can save
for this individual we make $498 on average if we don't offer the discount and now he's really gone from we made a prediction and then we didn't really have a way to stay completely what gets us a better fix your pennis values to now we're just deciding Morning for her $50 in this customer or do we want to make 498 if you consistently make a better decision in this case is about 10% more at the upscale. I've walked through I'll take an example that was related to
turn. Where is this useful to allow you to integrate multiple models and provides a nice layer to have to do this the place where we've seen a fusion and I felt so much for you to do it in today a lot more effort. So what are the most obvious Place pricing I gave an example about a hotel. They said they would like to know for any given price We Said Today once the impact it has on how many rooms we sell as a result how much revenue we make today? How does that affect spark of parasites tomorrow which in turn affects how many how many rooms tomorrow with
some friends are effects of rubbing tomorrow? How much will we make before the night of the steak and shrimp racing? We have I used to working with someone for fraud fraud prevention helping them do a better job of prioritizing which transactions they manually investigate fraudulent. We see a lot of people today have a simple machine learning model and they just save everything that we're going to every every transaction that comes in we score it at least by 5, so I could be fraud and we take some action. Every investigate it
if I have one. Thousand dollars is 4% likely to be fraud in one transaction for $10 is 5% likely be fraud will prioritize the $10 / that thousand-dollar transaction, It's costing you money instead of doing a better job of figuring out the rules for what are we investigate as fraud a lot of layers to that even Beyond accounting for the dollar value of the transaction. We see is a product recommendation. You probably should not be recommending just as big as soon as I could have purchased for quick on you should think about the
dollar value. You should think about which items are they buy lead to customer lifetime value of a customer loyalty that's against something that you can bottle see it in for equipment maintenance supply chain management. So you in about 80% Machine learning work clothes with cavities from excluding robotics, excluding computer vision in 80% of cases with traveling machine marlin fishing tables of data, you'll get much better business results. If you
formally model what happened not just in the machine learning model before only model what happened after that data scientist in the great focus on machine learning and as a result is unfortunately, I need to pay off to release taking this seriously is far greater than the path. We had eaten of the last few years of improving the Quality Inn in assets before after pictures. So happy and happy to answer questions. Anyone has about about this workload.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.