Duration 33:04
16+
Play
Video

Operationalizing Model Lifecycle with MLFlow By Anindita Mahapatra, Solution Architect, Databricks

Anindita Mahapatra
Solution Architect at Databricks
  • Video
  • Table of contents
  • Video
Request Q&A
Video
Operationalizing Model Lifecycle with MLFlow By Anindita Mahapatra, Solution Architect, Databricks
Available
In cart
Free
Free
Free
Free
Free
Free
Add to favorites
45
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speaker

Anindita Mahapatra
Solution Architect at Databricks

About the talk

If creating a production worthy model is hard, then managing the model lifecycle is even harder. There is a zoo of frameworks and tools that ML Practitioners can leverage.  Wouldn't it be nice to have a scalable framework to manage all these variations so that it can work  with any ML Library, Language and existing code! MLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. There are various flavors of deploying models to suit a wide range of use cases including batch, streaming, realtime, edge, ensembles and many more. In this talk we will look at some common patterns used in the industry to not just build but also manage and govern models.

Share

Plumber people I got to her wonderfully long and nice introduction but in short yet. I'm on in the time currently a Solutions architect at the date of bricks. My background is in software development Consulting excited and talking with all of you. If it were not school with times, I think we would have had a better face-to-face interactions, which is how we have been doing it for the last few years. But this time it's a great that Global Big Data conference is still able to host a virtually and today's topic is operationalizing modern life cycle using an Oslo all of us at some point have

experienced these Familia scenarios a data scientist created bags with JetBlue all the metrics, but it cannot be reproduced in the production environment that it's truly needed or it all works. So well during death and staging and you did manage to push it to production, but something went wrong the results around and we need to roll. Are these are real life scenarios and you should be able to address that because they do have it. Introduction to data bricks for those of you who may not be familiar with it. Apache spark is the de facto standard in the Big Data space and it was

created by the founders of databricks who contributed in Back to the open source Community select spark that you are other offerings. The database is open. So stingy with Delta V - animal slow about these are the things we dropping today is ml slow. It's because there are three main types of data personas on any data platform and databricks provides this unified platform to bring all of these person has to get so that we can collaborate and drive business outcomes with that data. So for instance those of us, what is that Engineers would leverage Delta more to get a solid

foundational basis for the bi analyst it's going to be Dash with all this dashboarding and visualizations and sodium of practitioner. It's going to be a male slow. So today's topic is a male slow and that's the one Play the focus on it was launched some what is a half years back and it is pink record numbers in terms of its adoption as you see on the screen here to million monkey downloads 200 plus gold contributor hundred plus contributing organizations. And that is the power of Open Source that we hear

from customers across every industry vertical. It does not matter if you're in healthcare or in the aeronautical space or in iot or whatever, they need some problems and what teams have no standardized their base interstructure and each one tries to build confidence that should ideally be reusable. And by the way, they're expensive be spent wasted Cycles on setting up their environments restrictions France translates to missed opportunities apart from cost implication, so there is no doubt that mature. Recognize the need for an

ml platforms to centralize their efforts which is why Google has come up with his GF Facebook has its that Facebook a leaner beaner and Uber has Michelangelo system. So if you're lucky to be working in these organizations and using it great, but the minute you leave it at that said because it's a proprietary system and the flow is interesting because it is the only open source machine learning platform for governance and lifecycle management. This is today's agenda will look at some common challenges in the envelope space. I what makes it interesting at what animal

flow is and what are the component it brings to the table to deliver on its promise to the entire ml community and we should have enough time to see a demo. I know a lot of you may have already seen this picture Google published in 2015 where it is trying to articulate that ml is fantastic. It's great. But it's a very small box of the ecosystem where you have to get a lot of the other things right before your ml can actually work and that heavily depends on a lot of data at attacks, like getting the right data says verifying yet feature

extraction serving. And then if you have all these surrounding pieces, well sorted out then the end of God is going to work. I know a lot of us at get a kick out of that little red box, but you know, there's a lot of non glorified work that happens around it as well. So what are these challenges MLS transforming all major industries, but despite the Prudential Building an elapsed is still complex. In fact, it is much more complex than the traditional software development life cycle and despite its potential. It is so

complex because it's so there's a lot of hand off and it is real and not just on the availability of data, but I'm sound reliable data so that the insights that you did right out of this data are defensible according to the three main reasons why and lytic projects fail our account of data silos to fragmentation and shortage of technical skills. The rich was a few years back and Dayton Lakes came into prominence where you bring all your data together you break down the silos you consolidate you see a

360 view of the So that problem is kind of thought you bring it all types of data structure than structures that is structured the fragmented and them to space to restrict the amount practitioners from trying out a bleeding edge tool if that is going to bring in a new level of accuracy or to put into your mother. So it would be wrong to restrict the to set that animal practitioner wishes to use what would be better is we standardize the process and provide apis and integrated uimm environment

to facilitate better collaboration. And once that happens the shortage of people skills part will also be addressed to a large extent right now. Everybody would like love to be one of those unicorn data scientist, but very few of us have all the skills necessary to be a good one. I like this punchline here data data everywhere not to drop in sight. So I'm trying to find on the word inside as an inside that is derived from the data. So do you have definite he's the new oil, but just because you have a lot of data does not mean that you have

something useful or valuable on your hands. You have to be able to translate it into insights that can drive your business to improve its Roi or growth or saw the second problem to harnessing the state that is an important thing just being able to have a data Lake and having the data consolidation is still not going to give you enough. We'd have to answer questions. Like can you justify that all the experiments that you did you choose the right model for production? Can you

justify that the What is selection process was robust so that there's no unfairness in the process like Saints and insurance industry you give somebody alone and you do not give somebody else alone, but they have very, just because a machine learning algorithm told you to do so, you could land into a lot of trouble that can you reproduce the model that absolute Fidelity a lot of our clients struggle with this because this is an experiment a student and an ox do it you may be able to get them. But if you cannot do another environment the same way then it's again a

wasted effort and modest. I like all other living and breathing artifacts at some point. They are going to get stale. So you need to be able to detect drift and replace it with a new version and sometimes people don't even know what version is currently in the production. So to be able to manage that entire entering life cycle of a model just like we do it in the code words to get has become like a defective. And everybody uses that to check in Decherd you do your job your revisions. There is something fails, you know how to roll back the same thing should also be possible in the body

artifact world because like cold which is living and breathing your inner artifacts are also long-lived artifacts. So let's take a quick look at you know a journey often ml practitioner. You have the raw data. You juice them ETL you featured as you train your score morning to debug and then this process continues because sometimes you get you have to update your features and sometimes you get some logs with your some errands you go back to your real job. You're more than just stand still and you have to retrain and this was the zoo of the ecosystem Frameworks that

we were talking about earlier and you cannot really say that this is the best to because there is no such thing. If any one of your used case, you would choose one or the other than a model runs, but then it has to be tuned to be official. It needs to be deployed so that it can you have all the monitoring and logging bells and whistles that needs to be Model Management and body doesn't exist alone. Some traditional does not work alone. He or she works in collaboration with a lot of other folks. So and you know, sometimes you build model ensembles, which means you defend your mother depends

on the output and it needs to be done or error and which means that everything that was earlier fitting into your laptop may not work so well today and then there's government so there's going to be all that stuff you have to be answerable for the interpretability of your model, which is even harder Challenge and then we get Sophos Decatur than you know, I'm just doing this like the kind of show you how difficult the life of animals are tichenor is it might look really, you know a urine from the outside but there's a lot of grunt work. Some of it is ml related. A

lot of it is taken during day. Tabulated and devops related as well and I won't even go to do an eyesight check and asked you to read the bottom line. But as you get into most sophisticated on model building exercises, there are other things to consider your job orchestration your dependencies your cicd your data dressed and you know, are you running on tram? Are you running Cloud multi-cloud you have it? So this is where ml slow comes in as the open-source platform to manage this machine learning lifecycle. So it's not getting

back flexibility to you to build a model of your choice, depending on your scenario, depending on your data depending upon your requirements, but it's the process of this machine learning cycle. What are the companies when we talk about ml slow what comes to mind? The first thing that petitioners do is the use of different types of modern architecture to run their expect. So just like in a science lab you ran experiments similarly here. Also, they have a purpose. They have a goal that this is the thing that they are off to her. And if you consider that to

be their experiment than France associated with that experiment each time this tracking component of ml slow is going to record your run. And so that you can go back in time later and quede it and what does it record? What is important is just not the modern artifact itself, but it's the code that was used to build it. It was the date associated with it's draining the configuration the dependent libraries, and of course the results. Then comes the projects are doing our experiments. We are running several several of them, right but then

once you decided that this was the best one for your particular case get in the given the circumstances you wants to be able to take that project somebody else for that. You need to be able to package it or reproducible translator. Sometimes the more disturbing is very orthogonal to the way I'm already was created. So for instance, you can serve like you can create a character model but you can serve it as a python version of it also because a lot depends on the consumers. How do you want you can see in the model. So here modest helps you take the same model

artifact but package it in different flavors for divers serving environments and then comes to registry a few minutes back. We were talking about how we don't even think about getting the de facto standard for your gold Repository. The morning was at we have to have a central repository that people can still be daughter back and it's related content be able to annotated all this should be able to come and Discover it because you know, if I'm already doing that trouble reading this what you have to reinvent the wheel and more importantly when you use this registry as your central

source of food for all your ml artifacts from this point, you deployed into different environments you manage to versioning of the model and so you can keep track of okay version 2 is currently in the staging version one is in production and maybe at some point. It's too. Well, I need to be able to promote reproduction. And if I did it goes on so this number is ever-increasing unless you decide to retire the model completely other uses no longer be quiet. Serving becomes another consideration and there are different ways in which she wants to be able

to serve it. So from the model registry the most frequent are the most prevalent way. We see customers use it is in the form of a batch or a streaming workload. So that means you get your data over a. Of time you have it in some form of a table once a day or twice a day or whatever it is. You do it in a bad mood or detest constantly screaming in like they from an iot device or from some other extreme system in a continuous fashion and ask the data is coming in. You need to be able to apply the model on the incoming data school

that produced the result make a decision or storage again for somebody else Downstream to Prince you so this form of either batch of streaming can be done on a distributor platform so that I The need arises so as more data comes to plant pumpkins jail, and you should be able to grow with it. The other form is real-time serving serving is most like TBI based so you can take it in a container and you can deployed into Cloud infrastructure which are inherently elastic that you can take it to a sagemaker and assures. Ml. You have tensorflow. You might have even

custom that serving and one of them that is provided write out of the databricks platform is the rest endpoint but this is only for testing purposes for the most tedious work clothes. We would advise that there is a benefit in just using a highly scalable VM and going over there. Now maybe same model and it controls different things in our brain. So what is your definition of a modern so model could mean the source code the version the hype about images that were used to train it and architecture so it could be like a a

deep learning model. It could be a random forest model so that the architecture the dataset the dependent libraries. So if you don't have any of these you may not be able to reproduce that model and then additional magazine that is also good. For instance any key value tea bags to help guide somebody later driving the model or using the model as to how to use it any additional notes who the author was when was it rain? How long did it take to train? So those are all the things that goes in what comes out of it is a metric. So it makes it could mean something like an accuracy true

positive rate of false positive rate. Precision recall those are the kind of things that you get out of creating a model and then you have the artifact itself. So what happens in the tracking server, which was the first ml so confident that he talked about is all of this is getting logged in the tracking Starbucks because what I did 2 weeks back, I may not be able to remember today but having a record of it it helps me to go back and search for it and justify that I chose this modern because this was the best auntie auntie Factory is read your store in the actual model itself. And

you do this several times. That is why that is these are called the train models of Peace Train models, you promote one of them into the registry which in your opinion. That was one of the better ones one. That was the best one. So now it from a train model it goes into a registered model that's in the registry in the registry. You give it a name of version of scheming against some dad's audit logs. And why do we do this? Because I'm from this point onwards the the strict guidelines have to be followed. For instance. The scheme has changed the function. If you are trying to look for older

version and try to give an example that images which is not expecting it might not work. So that is why it is important from the registry. You take it into a staging environment into a production environment or any other environment. So that becomes a diploid model and then you're going to retrain your model and you're going to read apply to other and other environments. But every time you have the discipline to take it from the registry and you have all these other checks and balances in place so that even in a in a production environment, you can see how what is what was the lineage?

How was it created? Can I go right to the source code and the other things that contributed to this this is the kind offer traceability that sometimes Auditors may be looking for so at this point, you know, we have said that governments has built in UK. Make a modern introduction and not only identify who created who approved what type of economy does better established to go to that particular version. Notebook that was using this is what I'll show you in our hopefully I'll get to it quickly. Let's look at what it looks like from a high-level let you know

that I wasn't with you. This is the image that I want you to carry with you because all of that other stuff will disappear soon. This is the whole slew of sophisticated Frameworks that the industry's constantly churning out on this side will be your customer your business needs a sample one in line code somebody wanting her as a shin Samuel 1 Baton streaming, you know, it goes on a good number of options on this site in the middle. This is what you have standardized so that you can take a certain flavor of the model that you have trapped with all its parameters metres

artifacts metadata. Take it into a staging environment promoted to a production environment benefits daily move from a version one to a version to take it into production. If that didn't do well remove it fall back onto the older one. So this is the kind of flu that would help simplify the life of a ml slow. And the promise of an ocelot follow us from there. It is open source is part of the Linux Foundation. It is open interface. So it does the FBI based and there is a CLI as well as a command line interface so you can download into your

laptop as well. It should work with them. Any of the libraries are Garten's languages Alexa Stinger code because it is wrapping around it is designed for scale. So you can use it in a single user environment or you can go to large organization and in conjunction with parking Delta, which are some of our Other Fellers it can support distributed after getting an insurance as well as flexible because people sometimes want certain language like it could be a python or r or a Java and you could use just one of those 5 confidants. I showed you like you could use just the tracking because

they are also Loosely coupled or you can leverage all of them for your end-to-end lifecycle management, and there's building integration with most of the popular brain books and I'll take you Set website in just a second. It does Universal. So which means you can use it locally that you can host your own a tracking server and use the apis or you can be in the cloud are you can be completely open-source or if you want additional support and you can use the manage version which is hosted on the databricks platform. And here you get the integrated you are and all the Enterprise Readiness

features. So I in shot and also is the open source projects. It's got the specs to see a Libras rest API UI and most importantly to Scott the community which means every day somebody is asking for a better future. Somebody's contributing a better picture and you could also run it on T-Rex, which is where I'll show you the message version. Animal flow. Org takes you to the main homepage and you get a lot of content here. But I do want to point you to the docks. If you go there you'll start to see all the apis listed under each of

these the individual components and then if you go to are a little demo for today, This is what I did a brick squad. Notebook. Looks like for those who are of you who may not have seen before on the left hand side. You've got, you know a way to access your metadata. You can spin up a plaster in a cloud environment. You can schedule jobs but a Vive focus on just some of the MLS extra day. So this is a psychic model. It is using a diabetes training data set is going to build a simple model and

these are the two important lines. We're staying in both and also an important as low as killer because we Escalon library and these are extremely boilerplate code where you know, you're drunk and creating some artifacts who you are and breaking down your data sets into training test and you're going to evaluate some metrics others root mean square error r m a n i a square. So these are the three metrics that we are going to compute and every time Train you're going to give it the same dataset. They're going to give it some type of guy to meters and it's going to generate

a model for us right as part of the model. We can either use Auto locking or we can choose to log data meters are cells so friends here. I'm going to log my bottom itches and go to lock the metrics and I'm going to log their model ASAP. If you want to catch your husband's if you have the Odyssey curve, right and that's very useful. You don't want to be able you don't want to go back in time and redo everything just to have the car so here for instance unlocking another artifact in this is just a p and she don't like an image file. So you have the flexibility to do all of

this now, we have our first run because all the train diabetes we have a date set and you call it that you have a better me. So let me take on this experiment app here. I ran this price and each time. It's going to create the serum. So I have my top-level which is an experiment and under that experiment. I have got three runs the first one this was with points in a 1in .01 so you can see my Alpine mile end ratio are here as a result of the modern running is going to produce these metrics with your here. And if I click on this button here, it's going to take me to the

experiment page. And if I click on this one run here, it's going to take me to the individual Tran. So this is what it looks like. If you go to the experiment page, right it's going to say that every experiment has a unique ID. This is where the artifacts are. I ran it three times. These are the three grounds. I can look at you each one of them in detail, or I'll say that let me compare all the three models if I do that it's going to ask me. You know, what are the battery does a what are the metrics that I wish to compare? And this is a very simple examples of three runs, but you

can imagine that if you have a lot of friends this visualization is very powerful because it will help you see your inflection points just because you increase one parameter doesn't mean that your mother is constantly going to keep getting better Bolivia point of inflection in which it starts to deteriorate. So those kind of things you can look and see very easily here. Let's go back. So that is one useful thing of being able to compare your models. You could also search for it like in the search part here. You can say give me all the models wear. My rmsc was less than

one or whatever. It is specific to your queso when you can you imagine you're having hundreds and thousands of experiments and quickly being able to natalynn on a certain set of experiments that repeatedly used for for you. Then if you want to go into an individual run even that to run Going to be like, you know reference by a unique ID. It will tell you the date. It was drawn how long it took but it was successful or not. The type of animal has used the metrics produced any additional dags that you may want to slap on it for future reference. And then remember the

second competent in the tub company which is around reproducibility of the model and treating in different flavors. So if I click on the model tab here, there's a Gamo file which is going to tell me that when the body was being created. These are all the dependencies. I was using this version of I was using this version of the site get the library and that all comes with you using that I can take it into another environment. The model itself is available in a pickle file and there are two versions of it. When is that because it's worse than SK model. So I

have the Australian version of it, but I have a very simple python function of it as well so I can draw. Where is the simplest form in which tomorrow can be drawn? And remember we were capturing the each of those the individual images for the ranch to even that we had long is an artifact and that is available. You can download it the model from this point. Like let's say this was your best model right from this page using the UI. And again all the apis are also there to do the same thing. You could register the model. So what does a model registering a model mean? So you

have promoted all you're the best trained model into the registry CNA registry of God all these models with a good motive to some extent and I have it with my initials here. And in the time of Akron this is my model. My latest version is version 2 and it seems like version two. I put only on staging and version one is a production surface left click on this and see some more details around it. In fact, let's go to my first version and see I want to promote it. I've done enough testing. Well, this is already in stage and just go to the second one. Sorry.

Let's go to version 2 This isn't staging and I say I want to request this to be moved to production. Maybe I give a comment nice babe tested verified whatever and I put in a request now if I have the Privileges, I would be able to promote it but this is like a workflow in veg. If you do not have the authority to push something to production you will get as a request now because it just me myself and I am the admin as a yes. I approve it and this is going to take to production as a yes. I approve it and I tested it and it confirmed it all of that other

good stuff here and this would get into production. Not useful thing is Amy audit any action that I do comes here and it tells like if it is a third-party order who's going to come and we can see a list of things that happen when they happen to approved it. All of these things are felt it and you can also enforces Kemah. Sorry for whatever reason the schema changes then you know that your mom. Breaking because of that and do you see like I'm in the registry now, but it is giving me a link to a run which actually produced the code so I can go I

can click on that and it's going to take me to that specific neuron that we were singing earlier and it said that he has the source code is also available to me. I'm going to click on it and come to a version. This was the second version of The Notebook with which I can go right to the code. So that is an incredibly powerful from your production environment you're able to see what was your drum version who created and all that other details and then you click keep clicking back to be able to get to the source code that produced it and all the other, you

know, any artists that have gone into creating it. So let me take you to some of the apis here as well. I don't think I'm doing too well on time. Is there a question answer session here as well? Or should I stop I can go on to digest and then and if you do a presentation that you are doing and I only see but example anything when someone says you have the model name and as I mentioned everything is that you that I did to the UI you can always do by API. So you say ml slow. You called tracking you

get the handle to the client and then you say Create a registered model, right and then you can choose that. I want to use this particular artifact of that model X in this case. I choose the first run and I said a created model version out of it and push it into production. And now that you have it and all the associated with the model from a particular stage Lexia production stage. It's going to tell me exactly what the model was. So in production on this time stamp for this run using this source

code link enabled Blinds To Go on the flash in the environment and you know, this can go on and on I can decide to take a different model and depreciates of the semi trosa the run to and I I showed you through that you I how it is to be done and this is showing you exactly the same thing but using the Apis to do so you get the latest version from production because you have upgraded now, you see that the first time it was the version one and the next time that you going there is diversion to there is one other part, which

is the serving side of it, which we will probably keep for another time. I think I've already overloaded you all with a lot of information for today. So thank you very much for listening to me and learning about the

Cackle comments for the website

Buy this talk

Access to the talk “Operationalizing Model Lifecycle with MLFlow By Anindita Mahapatra, Solution Architect, Databricks”
Available
In cart
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “Global Artificial Intelligence Virtual Conference”
Available
In cart
Free
Free
Free
Free
Free
Free
Ticket

Buy this video

Video

Access to the talk “Operationalizing Model Lifecycle with MLFlow By Anindita Mahapatra, Solution Architect, Databricks”
Available
In cart
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
566 conferences
22974 speakers
8597 hours of content