Kaz Sato is Staff Developer Advocate at Cloud Platform team, Google Inc. He leads the developer advocacy team for Machine Learning and Data Analytics products, such as TensorFlow, Vision API and BigQuery, and speaking at major events including Strata+Hadoop World 2016 San Jose, Google Next 2015 NYC and Tel Aviv and DevFest Berlin. Kaz also has been leading and supporting developer communities for Google Cloud for over 7 years. He is also interested in hardwares and IoT, and has been hosting FPGA meetups since 2013.View the profile
Rakesh Talanki is an accomplished Technology Architect with extensive experience in strategy, architecture and managing large scale high volume high throughput technical solutions. Rakesh currently works for Apigee a company that Builds and Manages APIs, Mobile apps and Data in a cloud setting. Strengths include deep understanding of APIs, API Management, Cloud Solution, J2EE stack, making sound technical decisions and participating in all aspects of software development life cycle. Experience building, managing and fostering strong relationships and communications with vendors, agencies and on/offshore teams. Proven track record delivering project initiatives to success with high quality and on time.View the profile
About the talk
The digital transformation in next decade will be empowered by what we call it as "ML/AI Service Mesh". Even though many companies are now generating features from raw data and extracting business insights with ML models, the challenge has been to share the valuable asset for internal and external consumption at scale. Each project or department in enterprises are siloed in the most of ML/AI projects; building features from raw data, training ML models, extracting embeddings, building prediction microservices, and use it internally. There is no standardized way to share the valuable assets and microservices with cross-functional groups and divisions.
API management is the missing link for building the service mesh quickly. By introducing a standardized and established way of securing services, enabling service discovery, and observability, Operations teams don’t have to spend much resources on exposing the assets to enable the ML/AI Service Mesh across the enterprise. This approach democratizes the ML assets for faster and scalable enterprise-wide consumption.
Solution: AI Platform + Apigee Edge
In this session, learn from a ML model built in the Cloud Machine learning engine and look at ways on how to consume this model from an internal consumer and an external consumer perspective. We use Apigee’s API Management solution to expose the models. This video also touches upon how to build "ML/AI Service Mesh" where enterprises can build a collection of microservices that exposes the features.
The demo provides:
- Serving predictions with scalability, performance, and availability in mind
- Authentication, authorization services depending on who the user is
- Managing the life cycle of API keys
- Granting access to your ML APIs with an approval process
- Rolling out new model versions as models are updated
- Self-service consumption using Portal without any DevOps involved
- Monitoring and Analyzing Analytics
- Monetizing the ML Models
Speakers: Kaz Sato, Rakesh Talanki
Google Cloud Next ’20: OnAir → https://goo.gle/next2020
Subscribe to the GCP Channel → https://goo.gle/GCP
product: AI Platform Training, Apigee, TensorFlow Enterprise Core; fullname: Kaz Sato, Rakesh Talanki;
event: Google Cloud Next 2020; re_ty: Publish;
I already want to thank you for taking your time for our session from ml to a note using API management, I'm cassato, Deborah podcast from Google Cloud base in Tokyo. And lucky Steinke. I'm Google Cloud architect and I appreciate it. I am in Atlanta Georgia. Invitation, we try to cover these topics as I direct you. Just get about how you can build a scalable machine running production system is Google Cloud platform. Then I will discuss about sikanderpur a problem that is called titled ML. And how you can solve that
by using an inside is mesh. In the latter part of this session rocket should be discussed about a PA management and how it can help building. And inside is mesh. So, let's start discussing about how you can build a Scarab in machine running system with AF at 4. Now, many data scientists or companies are trying to bring the PO system into a production machine running system. but those pills Co-op rotax can be recovered by the single data scientist, but the year for production system, you have to split the responsibility into the multiple roles
For example, data engineer can be responsible for ingesting, the debtor from data warehouse and a part-time employee processing against that way. Any of us can be responsible for 20 modular design. In machine learning model, get the higher accuracy, and they put the tranny motor into your system. And then engineer, or an engineer can be responsible for designing a scalable and continuous training, training, and production system. I think about that are you a
scientist use visiting and key people of the opposite? Things to ask to bring his deputies systems into production system. What kind of problems they often have to solve? There are many gaps between POC and production. For example, many people say notebooks can be local independent or dependent on a single person but with production system, you have to make everything portable unshareable. And that you cannot rely on the thing to know what the book is not anymore. You have to think about doing the class to her office. Multiple instances is just
getting out of skating. In fact, sonorities or load balancing all the incoming production request, And everything I should not be dependent on a manual process. Everything should be released in April usable for the Ender, to an amazing life cycle. A lot to me, you have to think about laundering so that oxygen can be responsible for the continuous monitoring and lighting for the year continuous operation. So do that, the reason why Google provides a product called while the airport one that has two features training and prediction. Those are my days to services for
machine learning training and prediction. For example, if you are using machine value prediction of the cloud platform, then you don't have to spend your time for building a Scarab oil production infrastructure by yourself. Instead, they managed service can take care of the adding more instances were removing, the instances were getting for the protection system. I don't so it can provide that it should be the GPU and CPU training service so you can easily get the certificate. The GPU environment without spending much time on building. A DP Crafter. I
don't sell the service, can provide the logging features are using the sap query data warehouse. So that makes it much easier for building a contest motor and feta. So, why you didn't grab the aircraft Forum, you can easily get you. Those benefits be quiet for a system that includes the scheduling of the year or instances lifecycle management of those continents in instances getting out. And it's getting ink on sonorities load balancing and dispatching. You walk across logging wandering troubleshooting and security, you don't have
to be rich those features formed from the scratch by yourself. Instead those features are already available in a art form. Bath Body, large Enterprises. That is not the end of the story. Stop example, Ritz. Imagine a scenario where you have just finished building at Mountain Valley pipeline for single satisfied. If it's a large Enterprise then you have to move on to another project like as optimization. Then you have to build another wall detection, demand forecasting,
customer support research, and so on. So if that's thinking about sharing any intermediate result or assets, you'd easily end up with having a multiple titles of matching body pipeline. That is an antiparticle titled ml. So poison apple for building a recommendation system, maybe you are getting two user profile user better from you the activities or product evicted from prokaryotic collection to bring a mother for the recommended for accommodation service. So the solution could be a building a machine. It's a business.
You can start me speeding among his kind of this service that has the everything inside it, but you can speak your responsibility into the multiple microservices. And eventually you would happen that can bees recomposed to build a final answer. Wake me up in case of a decomposition systems rather than building on the monitors all systems, but you can speak. Did Michael, Titus, Titus are, you can predict a user Victor from the youth activities, we can predict a product from product description that you could see
that can take us to also use a Victor and product, Victor to provide our recommendation result for the other purposes. So this is your solution call MSI. This message about the missing link here is the API management. And why is that? Because if you are working for a ride Enterprises, you have to think about how to share this. That'd be the proper business model and security model. You cannot expose your apis. Without any are the tracking and monitoring for that. The meeting is the API management for the MSI.
Business is getting more about API management. Thank you guys. So let's continue with the description webcast stopped. So we have them in pipelines. You want to introduce them as part of the Enterprise consumption. So how do you go about like you might have internal consumers must be for the MLP pipelines. It might be in town but you might also have the external consumption, the third party where you don't have activator, you don't have control. How can you take your pipelines or can
the other services and make them easily consumable by your consumers? Without any inspection, ideally in a cell service mode for this is where we're going to talk about and how to make it happen. Most of the company's most Enterprises and his journey moving from monuments in to Microsoft services. Like maybe some companies are halfway to to service-oriented architecture, but still going gone into Breaking Dawn, monolith into smaller services and made the communication protocols between
lightweight and simple. And also, many of the Enterprises are making the assumption that defrosted established between the services without actually enforce policies. For the idea of a good microservices, architecture is to increase. Like, you want to leave your features as often as possible manage complexity of the cold and you won't admit. It was Unity write each day as if their own to search. Maybe something's might like something, you might like. Net. Mac McKnight and
then you won't apply security policies. Very, very important. You won't do any birds that trust to Enterprise security policies. And then at the same time, you want to empower us to be as quick as possible and as independent as possible. The distance, the hall and Enterprise Services can look like you have multiple services in all. Communicating with my default apis becomes a communication contract between Services has some kind of different ways of doing a t. I, i maybe they're
using last night, so there's a measure of services within the Enterprise. And then when a consumer from outside of them inside, cause an API orchestrate, the calls, within the match, and get back with a response. So, we are getting into some kind of a mesh shirt. Let's see how to enable test next week later, but we have service. Now, these are the best place to have mutual TLS idly. You want to have certificates for each of the services and once you have unique certificates, you can enable the certificates and other
services, and you also want you to use a centrally describe communication policies, like you no replies timeout, circuit breakers. If a college, did you want any thing? So, they give the Services located in different regions. Different teams to enable auto rotation. Maybe not, every service is consumable by where Elizabeth search wedding Barn. You're authorized to services. You want collect analytics and also monitor the services. You want to know at any point in time are done. And at the same time you want to enable some kind of tell him to let
you know who is consuming what at any point in time. So we need to sort of a support system. Legacy. How to make this happen, this complication protocols to a standalone, something called Envoy Envoy is a high-performance pluggable poxy, providing Internet working and abs herbal tea for increase. The service communication. So, this is where some idiot box engine, X popular and widely used and it's very widely used inside Google also. Uses the Water Garden State Capitol in the cycle, pattern is better
than embedding different life and making it a bit more. So with this part of it is easier to support services implemented in multiple languages and also, you can update the boxes independent of other services. So this is where, so this is where a good control panel, help you to manage your services. So let me have the data scientist at this point. I'm thinking, it's just challenging, or is it simple like now a marvelous become part of a mesh architecture? It's no more than blank
of Martyrs are part of a big, big pieces, that device. Select start digging deep. Now I'll be wanting to use the control plane, which is a service match. What does service match help us to do? I connect between the tub and the floor 5 in between Services between the services and entered the communication. Very, very important. And you want to control this by applying security policies and then you want to observe and collect Matrix of monitoring and Analytics.
Let's dig in a little bit deep light still comes into play night. Still starts with the envoy foxy as a sidecar parts and then acts a set of policies to help manage and Soarin as you can see in this picture here, thank you have. The envoy has a control plane and then will also introduce what is called as if you match me to get here. The latest talk to this. So how when a PM I would help you and you want to enable them both for internal consumption and your extra consumption
and lipstick and I'll explain to you on how the API management help you to do this. So let's take this example already have a legacy app and you have a monolith app and you have the consumption there right now, and then most of the Enterprises have an API Gateway. In this case, a PGA event to my GPS enabled service in the back end. And then you have your internet laughs in most places into the last. I got to keep going to get his Services. There's not much of a PA management
going on in 30th. No latest. So go to this journey of Breaking Dawn, a monolith into Microsoft's at the same time you're updating Dental allowed to consume your Microsoft Windows as well as the monolith and at the same time you're updating you also use the service And in the process what you doing is your introducing new capabilities, by adding more microservices and also you using MLP pipelines. This is where you are adding more capabilities where you enter the laughs at being updated to consume
these new capabilities. And at the same time, you keeping your transparent for your exam consumption, to use the new capabilities. So this is, as you can see in the, the onset of qualification of different services. So this is a problem. How we can simplify this So this is where is Pure control, clean will help you to build a service mesh architecture. In this is kind of helping to manage the communication between the services that you also see the Legacy system, using the Legacy Services being being exposed to the gateway, to the third party system, right? And also
using the same time, a human services from the Thanks. And in fact, the gateways acting like a nice. If you switch to switch between the Legacy and the new owner of my magic picture, you can also see what is color is Applejack and this is where you get the API policies. Now what does that have to help you to do it hurt you to spend on manager UPS? It enables a publishing visibility of the API, find your best practices and it collects, specifically, from a security perspective of the adapter. When help you and foes in the envoy like things like verifying the water on the API Keys. Write a
chicken. If there's enough authorization to consume the services and other things like a quarter, if you have some kind of a quote on the service again. If you have at least limiting again. If you have electric protection to validate any place to do this kind of security policies and it is helping you to enable the communication between the services. This is kind of inverse equation. Of concerns is very clearly maintained. So that is the understand how this is happening.
So Envoy support a long list of Realtors and these Freighters and compiled into on what I pictured after uses a one special order external authorization, external outfitted on what decisions to a next-gen system. This is where I appreciate the gesture and hundreds of control. So, let's see how the traffic flow happens when a vehicle comes into a Envoy tank on what used to control the policies that are considered a Publisher's design for the Spurs play today. It can be cortisol production rate. Limiting
any policies that have? It's enforced if their response back with it, then if you go back to your back and service end of the cul-de-sac, buy sparklers near by another thing, back to your consumer and our does to Envoy dollar the APA convention, how does this work in a very high performance system? How can you get thousands of people in a bit, thanks for the first communication between me and the App Store and download anything that is required to make the decision which includes all the keys anything. Be quiet, all cast inside
the remote service. So this is how you can visit all the app. So let's try to bring everything together. So why use a pain management in your service mesh? Like so you don't want any of his services to be in a silo Lite, Whether tis your orders over, there can be a Michael service or I can be anything needs to be discovered and how do you discover them? So you can have a portal where you can publish all your services and enacting sense of the streets. There's no more. I just
think you're and then and then, and then you can modernize your services all, all the stuff on the Legacy, your monolithic equations. Everything can be hidden by neistat. And then no more, exchanging those PDF files, the word files that you can have nice documentation. Wherein, you give the required technical aspects, for the API, and the gym again, and before you want the music in application and then you can have the ports on the services. This is very, very critical condition. Be able to answer the question like a former top five consumers
today. What was my performance products to use for my business? So you can you should be able to answer these kind of questions. Using the color becomes very well informed that you want some kind of best practices in for sparkasse team and this is critical. You can enforce the thing sexy eyes. All these kinds of things occurred Enterprise, you can enable and enforce them, you monitor them. At any point, you should know which service is a bit service is going down before
you cuz you must tell you, which is very rude. And then once you have this nice that your system working, you want to start making money order and start charging you and this is where the monetization model and Define their policies and how to monetize APS need to be monetized. Maybe you pick up your business if you are such a critical, you only monetize a certain set of a kind of behavior. So you should be tied up all day and you should be able to try the production model in the tortilla platform and then
expose that are prime using the package engine and then send it to a consumer's. So all this, we have a solution to explaining all of this Mission will be. Once you have the smartest, how can you monitors are the services, how can you make your consumption of the services seamless unless you want to be the easiest of all, so that your services are actually continue lot faster, given the benefit to the Enterprise, like the name of this kind of action will be. How do you bring in your services in a way, it can be consumed with out any infection?
Look in the head. We have a solution page where you can try this sample example, where you can try the Air Platform and that you have to buy step instructions, 350 you get almost all the features in a Fastback form. You can find out. We have a very good Community. A lot of questions have been answered. If you have more questions, feel free to ask that. One of us will answer that. And if you need to reach one of us for more discussion, feel free to reach out by the cast. On me, will be happy to help you out.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.