Duration 39:40
16+
Play
Video

Personalize Actions for the Google Assistant

Adam Dawes
Senior Product Manager at Google
+ 1 speaker
  • Video
  • Table of contents
  • Video
2018 Google I/O
May 9, 2018, Mountain View, USA
2018 Google I/O
Request Q&A
Video
Personalize Actions for the Google Assistant
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
8.1 K
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speakers

Adam Dawes
Senior Product Manager at Google
Silvano Luciani
Software Engineer at Google

Adam has been a product manager at Google for over 10 years and is currently the product leader for identity developer tools. He has worked in computer security since 2003, building email security, identity and policy controls for Google and previously at Postini. Adam also led product development at Autodaq (now ADESA), a B2B e-commerce site for the automotive industry. He holds an MBA from Stanford and a BA in visual arts from Harvard.

View the profile

Silvano is a developer program engineer currently working on Actions on Google. He has previous experiences working on Firebase, Google Cloud, Google Apps, Google+ and AdSense. When he's not working you’ll probably find him spending time with his toddler twin daughters or cooking for them.

View the profile

About the talk

This session will cover best practices for personalizing Actions for the Google Assistant. It will walk through APIs that provide useful information to help customize the experience for every user, and demonstrate how to use authentication against existing accounts so the user experience is tailored to personal preferences.

Share

welcome to personalized actions for the Google Assistant thanks for being here and Savannah Beach and I work in the assistant and actions on Google developer relations team hi there my name is Adam. And I'm a product manager and we build identity tools for Developers so when I stop for an hour or so we're going to send you some of the functionality let the actions on Google platform provides to you to personalize your action I'd like to briefly talk about why is it important to personalize your action

technology that is at the center of an ecosystem of more than 500 sorry Millions devices I need to get things done things like the thing listening to some music or playing some games I got informed by reading of watching some news learn new recipes and so on Actions on Google is the platform that allows you to extend the assistant. You can have your own actions so that you can provide the help for the knowledge domain where you think you are an expert and you can provide value to the user you can connect with new users across all the devices that

we've seen and you can't even delayed by adding conversational interface that can make some of the tasks very easy. Just a straight voice command. So thinking about this that are two aspects where it's important to personalize your action. Number one is for some of the things that you might want to do, like for example, or send some music that a user can listen to to be entertained. If you know more about their preferences and The Taste is you can provide them with music that would be more in line with that expectation and that will fulfill during

Tan in a bad way that your actions can be involved across different devices. And so you want to be able if the user SS press some preferences on for example of a smart speaker owner of those preferences also when they start another conversation with your action on a different device like a van or is Mark this place where you will also be able to provide a better experience because now you can also show visuals to provide a better output to the user So we're going to look at three

main packet that I already almost mentioned. The first one is what is the platform provided to us so that we can learn something more about the assistant use it when we see how we can start this information in the current conversation that we're having with the user Orono the conversation that we might have with the user which means future conversation and also conversation on different devices about adding fraction that can be on time and in cases where for example of the use that

wants to know where the next payment for the certain service is do you need if you are at the service provider you need to know who they are on your system you need to attend you gave them so that you can actually retrieve the correct information for that account. So something from learning more about the assistant user. I'm going to introduce answering time. Comfort Inn pants are the core concept of the assistant platform. You can request to the assistant the momentarily take over the conversation so that he can perform some special tasks on your B are some

of the special tasks buy things like you can ask to transfer the conversation to a different server face with different abilities. If for example, you started a conversation on that smart speaker you want to show a video to the user you can ask the conversation to be transferred to a phone or the nadas device that can show the visual or you can even be playing into an Android app. For example, the user has requested a task that you think it's not it's not that he doesn't provide a good user experience with a voice in their face and you'd rather sit upon to your existing Andre.

We're going to look at a subset of the Albert that allow you to ask to obtain some information from the user. This information can be a Two different types one is you can ask to the use of the consent for the assistant to share some information with your action like the username or the device location the location of the device where the conversation is happening date and time and place and location in rather a way of asking an input to the user. I need the date for you. If for example your actions allow users to book services and you want to know

when they want the service or a place and location if you are arranging a delivery and you want to know why after that even the gods that are part of the transaction So something from the first one, how can we get the username in this life is what you see on your left is a simulation of a conversation that I did on the test app that I grew out and on the right you can see some of the code from AJ s not your library. It's not the completed for the action is just the base that are relevant to the alphabet that I'm

showing. So the first ask the permission the permission to get the name of the user and you provide the contact that is very important because with the context you are explaining to the user why you need the name. How is your service going to be better if you obtained that information? So in this case, I'm just telling them. Hey, I want to address you by your name. The assistant take over the conversation at the user. Can I share this information with the action if there's a positive reply then you get access to the name and

you can use it straight to use it by the name. It's not you need to deal with it and knowledge that. They didn't want to share the information with you. Try to provide the best service that you can. Disconnect life is showing to you the row Json API. So if you're not using the glad your library and instead you're using the conversation Web book, this is how you request permission. So you can see the intent name is action that intended permission and then there's a bag of parameters that can go they are different with forever

and they are defined in the body as fact that it's just saying this request my contained just type of field and you can see the contacts while I'm explaining why I want that information. This is showing you in the following request for clear to receive from the assistance. If the user consented to give you access to the information, this is how you receive it. Location is very similar because it's exactly the same intent. It's an intern that is called permission and one again. Once again asking the user. Can I share the location of your device with

reaction if the user replies? Yes, you got access to the location. And now in this case you could resolve the latitude and longitude for the place where they are and then find something that is near that location. Jason is pretty similar if you see if there's only like a different value for the type of information that we are asking and one thing that you can do this now is put missions is in a red, which means if you want to ask for both the name and the location at the same time, you can do it. You can just ask for more information in that array.

This is how you got the location if the user content to give it to you. Now this time is like the different as I mentioned because in this case we are not anymore asking the user if the assistant can share some information without action. We are directly asking the user. Can you give me a date a date value and the most powerful thing when you do this through the assistant is we can resolve ways of a specifying the date that I'm not just a date format. So if you look at the example, when would

you like to reserve the table? I reply tomorrow. Yes, please send results tomorrow to what tomorrow is at the time that we're having the conversation. Another thing that you might notice is in this case, we have three prongs does an initial from that is the reason why we want to ask that information, but what's up a date and time additional from that the system uses if it gets only parts of the information that you need from the user. So if I only specify a date like in the case of the example of what I just said tomorrow, the assistant will add what time using the

problem that you specify and once you have the information you can actually see it in the case of the note a ice Tanya Library. We represented as a not if they object Again, not very dear friend. This is the Jason. So what is changing is just the the bag of parameters that you are passing when you asked for this out there and the value that you got in this case from Justice and it's a little longer because you have to date fart and the time part. But I tell her that I want to present to you is the place and location Alpha. This is not

anymore asking the user. Can I share your current location with the action is asking the user give me a place give me a location again, you can see that if we can resolve things like the public name of a place so funny cuz I'm fall when the when the auction has what would you like to pick it up? And I replied Shoreline Amphitheatre what you actually get is the whole address so I can write on PDF PDF Parkway and so on. Not a lot of difference again. You already know what is API. It's all it's a

very familiar by now. The only things that are changing the type of parameters that you can send along with the Alpha and the name of the Alcatel. And the request in this case, it's you got the little bit more info because you also got the formatted address the public name and some more stuff. Why is no good? In this example where you at? I have side to side two different situations. The first one is the place we've seen all we can resolve a public place you ask for the Shoreline Amphitheatre. You just got the address but

it is showing something different is showing the location and location is something that a user the signs in math and its private them for example my home or my workplace and sew in the example on the right. When I asked what the assistant requested another type of concerned which can I share with the action? What work is for you? And only if the user says yes, then you can get that resolved address. If you want to see more about Elvis, we have a sample get up. Com actions

on Google flashlight of a flow Alpha Dean X not. Yes. It shows usage of all the LPS that are available on the platform. That's a couple of other information that can be useful that we can find in every request without the need of a new week of requesting any specific Alpert. The fun is just there a need for blasting and it's just the timestamp of the last interaction with the use it the first time they use it as an interaction with your action. It will be undefined. After that. You will always have the timestamp of the last interaction. This means that you can

use it for example to greet them differently. You can just say welcome the first time and then if the time stamp value of you can say welcome back because you know that they visited your action at least one and you can even do more complex stuff like you could calculate how much time has passed since the last time they visited you and the pending on whether you consider that range to be a short or the long time. You can read them and re-engage with them in a different way. The other one that is interesting for all of you that I have Android apps that allow user to

purchase or subscribe to entitlements is you can connect your action to your Android is Justin's equation in the console and verification that you actually only Android app and then all the entitlements that that use it as bought on your Android app and play will be available in a request. And this is an example of you will see it in this case. I'm just giving you a Json example because in the library with a sleep object that contains all this information. So this was it for some ways of learning

more about the user. What about starting information in the conversation? We've said that that it's important that we can do that so that we can give a more personalized experience inside a conversation but I was so across all the different conversation that the user can I have with your action? The first concept that I like to introduce to you is the conversation token the conversation. It's available only if you're using an action as the gate action and it's a field of the response and the request defined in the conversation Web book API. It's just that

we received you late. Like if you're right to it, we will stand back what you've written in the next request. If you change it we would do it again. The main cash is its value. It's always initialize to an empty string at the beginning of the conversation. So the life span is only the current conversation and it's all the values that you've stored in there will be clear when the conversation and so you can use it for some improvement improvement make sense only in the life cycle of the current conversation that you are driving. One example of

how to write a value in the first case. I'm just riding a simple string in the second case. What I'm doing is because I want to use a more structured approach and serializing a Json object to a screen so that I can get a representation that contains an object. An example of flow would you don't have access to the conversation token, but you have the same functionality Express with a more powerful abstraction called output contacts. And the reason why I say that is more powerful is you can have more than one out with context each one of the name and you can

identify them by name you can set the lights Tan in terms of terms of conversation for the weed that day that will be available and it's already provides an interface that gives you access to a structured volume. So to see an example of an output contact, you can see that help with contacts are an array in this case. I have only one but I could have more it's identified by a very long name the lifespan County 5:30 in this case and then I'm writing an object for a what are divided you that I wanted to start in this context. Do you use the guy the guy in the library or no? JS will give you

the best of both worlds because what we do is we provide you an abstraction data is just an object you write whatever you want to do the project and then depending on whether your attraction is an action SDK or a tidal flow. We do the right thing at the low-level using the conversation talk anyone case or the output context in the alley. This is an example of how you could use it. But say you have some music that you want to play that use that ass. I want to listen to some music you can ask them. Do you want to run them Sandra? Do you want to see if you can if they give you one one you can

save it so that for the rest of the conversation you can keep giving that music coming from that Sandra and Advance you say where are you count? How many songs for that? Genre you've been giving to the user and when they reach a certain point let's say ten songs. You are do you still want this? Jonra? What do you want to change? In some other cases the conversation Duncan doesn't work because letter you want to start the preferences for an action that is giving you weather forecast and you want to save the Z button area code identify the area for where they

won the the forecast if you saved it's in the conversation token. What happens is that value is lost at the next conversation. So you would be asking again, what is your zip and that's not a good experience and that's why I thought it might seem very familiar to you. It's exactly the same API of conversation token. So you already know how to use this. It's a field of response request is to string you can write just a string of serialized structured data will reset the lady across all the

conversation that you're having with the user conversation conversation on different devices. The main difference is the content can be cleared only by the app itself or by the user and we will see how to do it in a couple of lights. So this is exactly the same example that I asked for the conversation though. Again. The only thing that has changed is the name of the field now I'm using you and that means I have access to a beagle lifespan. Going back to the example of story in the Deep now that we started in the

user storage. If I started this conversation on a smart speaker save the zip the moment. I asked for the forecast for on my phone. The action will know the zip code and one need to ask for it again. That said this is how you clean up the volume you raise a Boolean if you're using the client Library set the volume of the storage to an empty object and we clear it for you. Last thing I have to say about the use of storage is very important. This is mostly like a cookie and so there are some countries that have strong regulations about

obtaining a concern from the user before you can say every day that from the use of storage. So if you're operating one of the stands with make sure that you use the consummation helper that is an alphabet that allows you to add something to the user and got the yes or no request before you actually start writing or reading later from the use of storage and with that it's not time for Adam to talk about that. Alright, thank you. Silvano. So user storage is a terrific teacher to be able to hold State and build continuity with the user across multiple

conversations. It works a lot like a cookie or HTML5 storage might in a browser, but we know that isn't always enough. So let me talk a little bit about how you can use identity to further deepen your experience with the user. Say the first thing that you get with identity is you get to know who these are is you get to know her name or email address and you get access to a profile picture. This allows you to build a direct relationship with the user where you can engage her via email outside of the context of using your app. The next thing that identity provides

is the ability to have a consistent experience across multiple devices and on different platforms. So with user storage you're able to keep application stayed in user preferences, but all of that data is stored in Google's cloud and only available to your conversational action, but with identity now, you can store all of the users data on your own back-end and whether they're not the user comes back via your mobile app the web or your conversational action, you'll be able to reset the state for the user and get her going where she left off the last

time. She used your service. And finally with identity now that you have all of her data stored on your back end identity can now help you secure that data. So only the user has access to it and nobody else does. City actions on Google platform has supported identity from the very beginning with of Base account linking. But the problem that we found is that one of user engages with your action via voice. The only way to be able to do that account linking experience with to punt the user to the phone and get the user to then go to

your website login and complete the linking experience. These kinds of cross-device flows are super difficult. And as you can imagine the conversion rate is pretty low. That's why we're super excited to announce the developer preview of Google sign in for the assistant it provides a way for your existing users or your new users to sign into your app completely via voice without needing to lie lie without needing to leave the context of the conversation and like Google sign in on other

platforms. We're actually able to automatically log users in if they previously used your service on a different device or another platform. Google sign in for assistant comes in two different developer mode in multi-platform mode. We layer Google sign in on top of a rock based account linking. This is what provides the seamless voice based account linking experience for Google users. It still maintains the path for users that registered with your service with another identity get a Hotmail account or a Yahoo account or something like that. Now for assistant only

developers, we're especially excited about what Google sign in for a system is going to be able to do we know that there are a lot of assistant only developers out there and we've heard you that identity and account linking is a heavy-lift just to get started on the platform. So now with Google sign in for assistant, you're able to rely completely on Google to manage all of your users identity. That means you don't need to build an account system of your own and it's totally cool if you've never even heard of a walk Let's take a look at how one of these

assistant only developers is planning on using the tool. 2 * butter a I was founded by for self-described personal finance key to love to argue with each other about the best way to manage their personal finances finances and look for financial services products. They all found themselves with a similar experience where their friends and families would inundate them with questions about the best credit cards to get or how to get a mortgage. It was through this experience that they realized that the best way to recommend Financial Services product

was to get to know the user through a question-and-answer kind of dialogue. This is why they founded the company and our investing so big into a I and our building on the assistant platform and they're doing great. Just last year. They won an award for being one of the top actions on the assistant platform. But the offer the kind of personalized advice that will keep users coming back again and again and that's relevant for them * butter really needs a user's identity last week. We gave them access to Google sign in for the assistant. So let's take a look at their demo

app to see what they're thinking about in terms of getting set up with this new feature. Talk to mortgage helper demo. All right, getting the test version of mortgage helper demo want to compare Loan offers for calculate how much you can borrow? Calculate how much I can borrow? You compare Home Loan offers. I don't need your basic info to create a new mortgage help from Google. Yes. Alex I create a new account using your Google account All right, wasn't that so much better? The user was able to create an account without leaving the context of their conversation and starve said that

it was super easy for them to get set up. So let's take a look behind-the-scenes it what's going on. Set the first thing to know about Google sign in for the assistant is that it operates like Google sign in on all other platforms the same principles apply where you want to get information about who the user is Google presents a URI for the user to make a decision about whether or not they want to reveal their basic profile info to your out. If the user says yes, then we generate and ID token and hand it back to your client. Now, let's take a look at one of these ID

tokens don't ID token is just a Json web token and that's basically a dictionary of Json which is cryptographically signed by Google so you can be sure that the that the information inside the token is authentic. inside the token you'll see all of the information that you need in order to be able to make a decision about whether to link the account or create a new account for the user so that includes the user's name their profile picture and their email address as well as the Google specific identifier that's unique to that user you'll also notice at the top

of the screen there's the AUD field that's short for audience this is also a super important field because that protects the security of your users data its what prevents another app from getting an ID token for it for that Google user and replaying that token against your back end to get access to that users data Okay, let's take a look at what star butter had to do to get this set up in their action. So the first thing that they needed to do was to go to the actions on Google developer console and go to the identity and account linking section and there they can figure that

they use Google sign in as they're linking type. Next they need to update their action. So first they need to initialize their client ID. This is what comes back in the audience and protects the users data and then they need to find the right time to be able to ask the user to login and provide consent to give their account information now star butter did that relatively early and you're going to want to be thoughtful about where you do this. It's generally not best practice to immediately ask the user to sign in when they open your actions. So think

about that carefully And then finally the last thing that star better needed to do was to be able to handle the resolution. Now this can break two ways. The user can agree to provide their profile information or they can decide not to if they agree then we'll be able to return that whole set of Jason that comes back in the ID token and then you'll be able to extract the user's email address or their Google identifier and then you can start keying data on your back end based on that user's identity. If the user decides not to provide

that information, then you're going to want to handle that gracefully. You're going to want to be able to explain to the use of the benefit of logging in and if they still decide that they don't want to share their information. You're going to try to forward the conversation and give be useful to the user even without knowing who they are. Okay. Now let's shift back to the other developer type the multi-platform developer things are a little bit more complicated because they want to be able to support two types of users both the Google user as well. As any user who may

have registered with their service given another identity provider like Microsoft or Outlook. So for them, they need to be able to have that path to be able to connect to data for both user types. Now. I understand it may feel like a little bit of a cough that only Google users are be able to get this voice based account linking experience, but I want to stress that it's still going to be super impactful for your action. We know that you've invested a lot of time and energy to build a great experience for users. And so we want to make sure that we bring as

many users through that front door. And so by having Google users get this streamline voice experience that's going to bring many more users to enjoy all the benefits that your action provides. So let's take another look at an action that is on multiple platforms. So this is a pitcher and their goal in life is to be able to answer that age-old question of what movie or TV show should I watch tonight? So they've built a personalized recommendation that's available on iOS Android web and the assistant platform

and they've offered many different ways for users to sign in via Google Facebook username password even Twitter. So now it's pretty self-evident that it's a really needs a user's identity. They can't store users likes and dislikes and make meaningful recommendations without being able to have a consistent experience across time and across devices. So, let's see how they're planning on using Google sign in for assistant with their demo app. Talk to itcher. Let's get the test version of a picture

and profile picture from Google. Yes. So now I can check to see if you have an account with us. Just a sec. I found your account Alex. Your account is now linked to Google TV show. All right. So that was also a much much better experience that in the background. We were able to detect whether or not Alex already had an account with pitcher and seamlessly make the experience. So how does itcher create that experience and other multi-platform developers with Google sign in for

assistant the first just like Star butter, they need to go to the actions on Google developer console. But in this case, they pick go off and Google sign in as the way that they want to do account linking. Next big you exactly like Star but are they find the right time in the conversation to ask the user to sign in? And then this is where things get a little bit different sew-in that interaction we went and we went to do a discovery to whether or not that account already exists. So in order to make that happen star butter and in this example, the the food bought at they

created a separate API a separate rest API from their conversational action fulfillment and point. So we call this endpoint the token endpoint and it allows us to do account Discovery and creation. Set the way the logic of this endpoint works is pretty straightforward. So we ask the user whether or not they want to sign into the app. If they say yes, then we generate and ID token and we send it to the Token endpoint in this case valid acid token, and then they extract the user's email address and do a look up on their account database

if they find that user already exists, then the link the account by taking the Google identifier and storing that for the user and then they'll also generate their own credentials and return not to Google. Google store that credential and then return that in every turn the conversation back to the food bought app, and that's how it will be able to keep stayed on her. The user is now if in that look up on whether or not that a user already exists that user doesn't exist then food. Will return an error saying I've never heard of that

dessert. So then what Google does is we look up food bought settings and see if they want to support account creation via voice not all apps want to do that. Sometimes the registration process requires that they gather more information than just the name and email address or maybe that they want to make sure that the user is really evaluated their privacy policy and terms of service. But if they do decide that they want to do voice based account creation, then we'll ask the user. Do you want to create an account with food. If the user says yes will then again hit this

token endpoint and the token endpoint will again validate. The request is coming from Google and then extract the name email address in profile picture and store that information as a new user in their user database and then they'll also create a token and return that to Google to store and then respond with every turn of conversation to keep that context. Now if food. Decides, they don't want to do voice based account creation or the user decides that they don't want to create a new account. They want to go and log into their existing account. That might be

based on a Yahoo or a Microsoft identity. Then we fall back to the regular a walk face to count flow where we pump the user to the phone and they manually login via their website and then food. Returns are credentialed back to Google this ensures that all users are able to complete the account linking flow and all of the Google users have been streamlined experience. So that's pretty much it. It's really easy to take advantage of Google sign in for the assistant whether you're an assistant only developer or a multi-platform developer. The developer preview is available

today. Just go out go and check out the actions on developer actions on Google developer docs and look for the identity section. Now as a recap silvano talk to you about how you can build a more personalized experience with user by using Helper and Tents to get permission to get the user's name location and time. He also talked about how to use in the request info to see when the user last use your action and whether or not the user is already purchased info from your surface via the Play Store next

about how to store information and build continuity with the user using the conversational token as well as user storage. And then I led you through how you can use Google sign in for assistant to get the user's identity and email address and the power of voice-based account creation and account linking and for his actions only developers now, they don't even need to have their own account system. So we know your time is very valuable. We would love to hear what you thought of our presentation. So, please navigate back to the io schedule find

the session and give us a rating. And if you want more information, please go online. We've got helpful. All of Doc's that describe sylvano's experiences in the best practices, and you can learn more about Google sign in for assistant, and the developer dock section will also be across the way at the assistant codelab and and an igloo tent, so, please come and ask us questions. Thank you very much and have a great last day at IO.

Cackle comments for the website

Buy this talk

Access to the talk “Personalize Actions for the Google Assistant”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Access to all the recordings of the event

Get access to all videos “2018 Google I/O”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “IT & Technology”?

You might be interested in videos from this event

September 28, 2018
Moscow
16
173
app store, apps, development, google play, mobile, soft

Similar talks

Naveen Agarwal
Federated Identity Team at Google
+ 3 speakers
Luke Camery
Product Manager at Google
+ 3 speakers
Treffyn Koreshoff
User experience researcher at Google
+ 3 speakers
Steven Soneff
Product Manager at Google
+ 3 speakers
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
David East
Developer Advocate at Google
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Jenny Gove
Mobile Payments UX Lead at Google
+ 1 speaker
Brandon Herring
Software Engineer at Google
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video

Access to the talk “Personalize Actions for the Google Assistant”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
644 conferences
26393 speakers
9799 hours of content
Adam Dawes
Silvano Luciani