Ilya is a software engineer on the Actions on Google (AoG) team, leading the efforts to integrate Android apps with AoG and the Google Assistant. He also leads the AoG Integrity team, responsible for maintaining a high-quality developer ecosystem. Ilya has been at Google for over 9 years, most of this time on the Google Play team. Ilya has an MBA from the University of Wisconsin--Madison and a bachelor's in computer science from the American University in Bulgaria.View the profile
Luv is Product Manager on Google Play responsible for improving discovery and re-engagement of apps by understanding their capabilities, attributes and content. Prior to Google, Luv worked at Bain as a management consultant in tech practice. Before that, he worked at Broadcom and Intel managing smartphone system-on-chip products. He has 20 issued or pending patents. Luv has an MBA from UC Berkeley, MS in computer engineering from University of Illinois at Urbana Champaign, and Bachelors in EE from Indian InView the profile
About the talk
App Actions can help you increase discovery and reengagement for your Android apps by allowing you to signal their useful capabilities and content to users of the Google Assistant, Android launcher, Play Store, Google Search, and other surfaces. In this session, you’ll learn how you can get started with App Actions - from implementing actions.xml, to understanding core UX principles, registering for built-in intents, and providing inventory.
My name is Loca Terry. I'm a product manager in Google Play Store. I'm responsible for improving our understanding of Android apps the content the capabilities and their attributes and knowledge with great experiences. Hi everyone. My name is Celia Furman. I'm a software engineer on the X on some Google team matching some Google offers apis for developers to integrate with the Google assistant and other Google products. My focus is on bringing Android apps into the actions on Google ecosystem. So
today is super excited to give you a sneak peek into app actions as they mentioned in the keynote this morning app actions will allow users to discover and re-engage with the capabilities and content from your apps in just the right moment. And as an Android developer, you will get an opportunity to get surface across multiple Google and Android surfaces and improved engagement and reach for your Android apps examples for this. Without actions users will be able to get things done quickly and
seamlessly with your friends. Will it be great if you had a personal assistant which could help you accomplish tasks such as making a dinner reservation getting movie tickets getting a ride to the venue. I'm building a collage from the pictures taken during the evening. A Google Assistant was created to help users with these daily tasks. And for many of these tasks that use them I already have the perfect Android app under devices. Without actions your Android app can integrate with Google assistant and participate in this user Journeys. Moreover is a passion suggestions
will also appear on other Google and Android services to help users get things done and discover. What did apps can do. Directions to be able to go from the content they love. Do the actions that can take on that continent? two development apps to fulfil those actions play Imagine A Lady Gaga fan is searching for the latest hits. Will it be great if that user why is hitting for the information to listen to all songs videos find tickets for upcoming concert? And find updates about her. This is the ideal woman to
discover capabilities from the Android apps you already have. I've actions can help users re-engage the app they forgot they had and also discover new opportunities and capabilities. They did not know existed. Good, we will be bringing similar experiences with diverse content such as TV shows movies books courses Sports Games books and more. As an Android developer, you know that re-engagement is a challenging problem. users have hundreds of apps on the devices to choose from
Android authority do they serve a couple of years ago and found out that 77% of Android users do not use an app after 3 days of installing it and that number goes up to 90% after a month. You all have worked really hard for those app installed but getting users to keep coming back it up is a hard problem. No, notifications and emails are all tools in your tool kit, but it's also possible that I can come across as Tammy and annoying. It's much better to re-engage with the user in the right context and in the right moment.
So today you're going Beyond simply protecting the next app that uses would want to open to the next app or next action user would want to take. Nai first world we want to assist users to find capabilities and content from your apps. Even if that information is very deep inside the app. So, how did you make this happen? First rebuilding Rich catalog of actions intense is it called them? Essentially think of them as workers known these are the ways you will find
what you're after capable of some examples play music play game. Take course or the right order food. These are all intentional catalog. We wasn't deep semantic understanding now, what do I mean by that? We will Define detail schemas with Premier specs and also Define grammar, not only in English, but many other languages. What is a concrete example of checking air quality in Los Angeles? The multiple ways a user can make that very some examples that I care check pollution in LA Smog levels.
What's the forecast for air pollution and many of the possible ways a user can make that very Google help at all of these queries and made them into one common Ben 10 and then call check air quality will also Parts such as location being Los Angeles and Time Peter being today. All of this will get them passed along to the Android app, which can then take the user to the right place inside the app and fulfil that action. As I said good also defined grammer's in popular languages and provide natural language understanding.
What's really powerful about this is that you get instant Global reach without actually having to do the hard work of understanding language is providing semantic understanding. We already support 60 languages and will continue to expand. Coverage. How do you participate as an Android developer? You just need to add an action selectable file in your APK? In this file you need to register for the intense that you have is capable of and it support and tell us how to fulfill those actions while deep links exactly.
Once you're exposed to capabilities and content of your house. Use Aquarius in context to your app and service you in multiple highly use Google products starting from Android and with an Android P launcher and smart text selection the Google Play Store. the Google assistant and the search app actions are not restricted to Android p and are available on all devices with the most recent versions of these apps. Let's talk to a few examples really amazing features about really hard for the last few months to put together.
This is my screen on my phone. They all apps and all that screen as you call it last year introduced the concept of predicted Apple. This is the operating system trying to predict the next app me as a user would want to open up. It's been working really great as they mentioned this morning we have about 60% prediction rate on this. It's phenomenal. But this year if you want to go further, we want to go from predicting the next app to predicting. The next action user would want to take. If you take a closer look into action suggestions and right around this time of the day
typically commuting back home. I'm relying on Google Maps to navigate me to the Breezy Casey Bay Area traffic to deal with around here. movie theater strategizing on how to survive with two kids at home Both of these actions take me deep inside the app to fulfil the. Immediately. Reactions are predicted based on usage Gardens routines sequences and also only contacts. I'm in my car, but I have my headset on cuz I'm doing another example if I really plug in my headset in my
my phone, I'll get an option to resume an album or playlist. I was listening to on my music my favorite music app. Actions are also integrated in Smart Text selection in Android. So that's a good recommendation to a new restaurant in the city of flour plus water around on the text Smart Text selection will recognize that it's a it's a restaurant named. It's like the whole text. I'll also try to map what possible actions and then further what possible apps can full figure action for the given entity. So as I
as I select floor plus plus water I get a recommendation to use OpenTable which is an app that I use a lot to make reservations on my phone. I can tap on that and take me directly into the Open Table app where I can see see more information read reviews and make that dinner reservation. All of this is getting powered using a non device machine learning model on the screen never leaves my device. Everything is secure and fast. Martex elections Newell network will be able to recognize
TV shows movies at least musicians flight numbers and more. actions would also service in Play Store And what's really phenomenal about this is that we all know there is an app for everything. And the Play Store is D place where users go looking for those apps and those capabilities. Search for Lady Gaga to go inside the inside any of these apps and fulfilled his actions. What's interesting is not only I get a tax considerations for apps I have installed but also the apps I have not installed. This is the universe of app that exist out there. If I tap on something that is not
install I go to the details so that I can install the app and when I tap open and I can take me directly into the right place inside the app to fulfill that action. Without actions users can discover new ways to re-engage with the content they already love. Actions will also work in assistant assistant to show me my budget on mint and pasta to show budget action inside the Android app and again the right place inside the oven get involved and I'll be able to fulfil Direction.
Here I'm asking assistant. What is Lady Gaga's real name is will understand that question answered back with the right information. But the assistant can also anticipate I might be interested in knowing more about Lady Gaga and similar artists. So I guess you an action such as red all I can listen to her songs. Watch our videos buy tickets for upcoming concerts back to the example. I showed earlier earlier in the presentation. It's a great way for users to discover capabilities about the apps you already have and they're able to do so in the right moment. Again stopping on any of these
ships will take me directly into your Android apps and then will let me fulfill the action of us intended to do. Finally has Dave mentioned this morning in the keynote. You all saw this we're exploring different ways to surface actions in Google search for apps that uses already use a lot as an example. If you search for the latest Avengers movie Infinity war in addition to regular suggestions. You will not get an action to Fandango app to buy tickets as you mean Fandango is an app that use a lot and you have it on your phone. really cool changing your
actions actions as part of a broader actions in Google developer platform processes are a set of building blocks Foundation Building Blocks, Early. An example of this you already saw in 10 catalog. This is the Texana me of some built-in intense for which will provide deep somatic understanding. It's structured data. You can provide Google the inventory for your actions. You can also report Action usage via Firebase app indexing & get recommended in personalized suggestions based on contacts and usage patterns. Identity and seamless digital subscriptions allow hassle-free
transition across multiple devices on TracFone factors and modalities. The tumor that is available. Number one Android app actions. That's what we've been talking about here. There's another one with this conversation actions and you've heard quite a bit about it in consumer and The Velveteen. These are the ones for YouTube invoice power experiences. You can also try nowhere to go program in deeply integrated with Google for and has the discovery across multiple Google services. So
Google is defining a rich catalog of intense with Steamers in grammer's as you sign up by providing us an actions or XML file where you can register for intense that you have support. You also tell us how to fulfill a sentence while deep links. And once you've done that you get instant increase in reaching engagement across multiple, Google and Android services. Can you walk us through some of this and show us how this is working? Thank you so much. Love. I hope you guys are as excited as we are about the opportunities offered by
app actions. You must be wondering. How do I sign up? Let me walk you step-by-step through the process of setting up actions for your app. At the core of actions is actions that XML and you can figuration file that you can include in your Android APK actions that XML enables you to register for intense and to Define fulfillment for those sometimes. Not when it's about intense in the context of action some Google, where's the most romantic intense? These are defined by the experts in the Google platform describe what your action does in the language the Google can understand?
Intense going to be built in coming from the rich catalog haven't answered love mentioned earlier or custom intense 2:35, but a developer. And Wellington describe what connection does fulfillment describes how your app does that action for Android apps fulfillment connects the semantic intend to the Android in the app? Together in dance and fulfillment Define actions and actions with XML is how you provide actions in your Android app. Once you define your intent and your fulfillment, you can help us personalized excellent suggestions, but integrating
usage login into your app using apis from Firebase app indexing. You can also extend your reach Beyond Android they build a conversational actions and you can enhance the way actions are presented to the user using slices and uui frame for framework for servicing deep enough content. This is just a quick high level overview of the FX and setup process. That does not walk through each step in detail. starting with intense is love mentioned earlier actions is built upon a new catalog of semantics and tents that are published by Google as part of the action some Google
platform for reaching temp. Would you find a grammar that helps us recognize user queries and parameters that we can infer from the query and pass it to your app. Ben 10 catalog help us organize the world of actions so we can show the right actions to the right user at the right time. We expect that most developers will register for one of our bills and intense. With built-in intense you give Google a deep understanding of what your app does. Moreover wood building in town. So you don't need to worry about natural language understanding. I mention after I owe you might want to ask your
system to get it right. There was a full or book a taxi to San Francisco Airport or to order a captain, San Francisco Airport. What building in France you don't need to worry about all the different ways in which a user can express the same intent. Google takes care of natural language understanding your app just needs to provide fulfillment. Built-in pants give you support for many languages right out of the box and they provide a one-stop-shop for getting access to multiple DUI services. We will be adding more and more intense into the Catholic overtime. So if your particular
Justice is not covered chances are it will be soon and you will have a way to request new in town still be fine. And if your juice case is truly unique, you can define a custom it down and build your own grammar using Epsom Council the equivalent of play console for actions on Google Developers. It is important to distinguish between semantics and pants that are defined by actions on Google and Android intense a feature in the Android framework. Etc developers, you're probably familiar with Android intense. They describe specific Android operations in a way
that allows Android apps to talk to each other semantic intense on the other hand larger catalog that extends Beyond Android includes the assistant Google search and other services. For Android apps actions that XML is what defines the mapping between semantic intense and specific Android intense in your app. Of course, you can build multiple ways to fulfill the same semantic in Santa Cruz different types of devices. For example, let's say you're a bank and you want to support the check account balance in town on and your mobile banking app is a great way to support content. A
separate that you can build a conversational action so you can support a voice only experience on Google home or smart displays using the tools that you've seen earlier in the actress in the Google Talk and you like to learn more about how to transition from building for your Android app to build a conversational experiences. You can come to a session dedicated to that on Thursday morning today with Android apps into the app actions ecosystem. Know if you're an Android developer, you probably already familiar with intent filters and they did mechanism in the
Android to expose intense outside of the app intent filters Define conton types and date are your eyes the traffic except and those days at your eyes and come in the form of HTTP or else or perhaps you or else in your own custom a specific scheme. Action physics amount Defiance. How is Samantha content inferred from a user's query or the user's context can be converted into a URL to open until in your app. Android intent filter logic and Converse that the URL interspecific Android intent to launch in your app Why do we use your else but the next two mannequin Townsend Android
intense? Deep link URL is a part of a larger F Discovery ecosystem. You can link your app URLs. Do your web content using verified app links or Scythe Maps? Will Google structured data programs? You can precisely Define the specific content that your app can support. Android Firebase app indexing, you can enable Google to personalize action suggestions, but by identifying what users do when they use your app. All these different features and programs use URLs to identify content in your app. Don't talk about intense
never switch gears and talk about fulfillment fulfillment describes the specific mechanism to invoke a particular action in your app how to determine the specific deep link URL to open for a given semantic content. With actions to maximal you can use one of two models to provide fulfillment for a semantic content. In the oral tablet model we use the parameters of the semantic intent to dynamically construct a fulfillment URL at Sante Aesthetics to melt provides a URL template with placeholders for specific parameters. At
one time we can take those parameters for the user in town and have them into your url. This model is ideal for action center caps with deepening apis. In the Compton driven model we discovered the Fulfillment URL throw your web content or your structured data. We use because there's a query to find a relevant content in your web presence and reuse action for XML to connect the Euro other content with the appropriate content. Of course, you must verify ownership of your Wisconsin before you can use it in your out. You can use Android as verified app
links or connect your app with your website using the Google search console. Oh that's really needed for Compton Drew and fulfillment is Lincoln your actions that XML to your web pages, but the content and fulfillment model works best with structured data. Why was Samantha contams describe the verbs that your app supports structured data and describe the nouns the inventory of your app? Google provides a variety of structured data programs. Some of them are you supposed to be defeats many of you probably already participate in these programs. You can use their structure
Forsyth to see if there's a vertical program that fits your specific use case and you can also find the length of the tool that you can use to test your mark up. So how they actually submit your actions of XML to Google. Perhaps you're developing your app in Android studio and you can include your actual text to mail in the same Android project and the same APK that you submitted to the play console. Play console Ben Parsons your actions of XML and registers your actions actions on Google database. This registration allows your after respond to queries and appear in suggestions in various
Google products. And how can you test that your actions work before you submit your app to play? You can use our new plug-in for the Android Studio gold the action step stool with a stool. You can validate your actions of XML and submit it to action some Google in the special preview mode, which make your actions available just for your own developer account. You can tempt test how Google Assistant would invoke your app with different combinations of semantics and pants. Cincinnati Contemporary Motors Let's take a look at some
demos. In his first example, I will show you how the Ural tampa-based fulfillment model works. I will show you my favorite text. Yeah, that's a build a couple of weeks ago for the session integrates with the order ride building in town for just one moment. switch to slides Thank you. That were to write them down, before parameters for pickup location drop off location pick up time and drop off time is parameters is defined as an entity. And you can see on the slide how we parse the parameters of identity from
a user on vacation and now this week. But you can see on your screen is Android studio and in my Android project, I have my actions of XML file open. You can sit here. I'm register my taxi app for their order right intent. And I'm providing a URL template. In this case. It happens to be an HTTP oral, but you can also use in a few specific areas or indents your eyes. In this case, they're placeholders in the Euro template for two parameters for the drop-off and pick-up
address. These parameters are the map to the parameters of being temp placeholders. I'm at the Brandon will be in town. Now, let's bring up the FX on Festool. When I click the load actions button there to validate my actions of XML parse it and submit it to action some Google in preview mode so I can test what would happen if I passed a particular parameter? when I click from oops For some reason my phone decided to switch into landscape mode. Okay, I think I think I made the point the
reciprocal of somatic symptoms with particular perimeters. More specifically and extremely and you might have seen excellent earlier today in the absence of Google Talk Samantha contains India system. Let's watch the flight, please. But something about the template baseball filament model now, let's talk a little bit about how the conference room model works. We have worked with Coursera to integrate their app with a pexels. Coursera signing up for the take course in town from our catalog decent time today takes
a single parameter, of course and you can see how weird question of course from a user on vacation. Now Coursera happens to annotate all of their course webpages was structured data. We can infer from the structured data what the name of the courses what the course is about a specific deep link URL for the course is and since Coursera is using verified app links. We can use the same URL to open the Coursera app on the device. That's supposed to. Here is the actions of what actions like some Elekid look like for Coursera. Grocery sign up for the take course in town and for the chorus parameter
of that then there is an inventory reference an entity set reference that ties this Intel registration to ski or course structures located on these web pages. And the Euro template in this case just take the Euro discovered from structured data and as a referral program. Again, we can bring up the app action step stool. submit action center in preview mode And let's see what happens if I wanted to study machine learning. something again at the Coursera machine learning course. Devil girl smiled us perhaps we can also see what would happen
if I try the query. What is deep learning? Hear Google has given us a son is given as a knowledge answer explain what deep learning is but underneath that answer. There's a suggestion to learn more by taking a course in Coursera. Am I click on that link? It opens a link to a course in the Coursera app? Wesley Snipes lights, please So you've seen how you can Define your intense and fulfillment using the actions with XML configuration file. You can further help us discover your apps. You can help you discover your apps
using Firebase app indexing apis. Action logging through Firebase app indexing Health Google provide personalized suggestions based on the use of context and usage patterns. For example, if you open listen to jazz music on your evening commute, let me show you a suggestion to play cool jazz radio when you get into the car or if we know that you keep watching other carbon, we may give you a helpful suggestion when the next episode comes out action log in and help your a Frank a better induce personalized action suggestions. This is
not a new API. We have made changes today that I specifically put Support app actions. You cannot lie each reported event to specific semantic in town and specific parameter values. Of course you remain in control of their data at all times. They can see and control of their load activity using the myactivity website and they can opt out of action login all together. Alabaster fish to love who will talk to you about how you can enhance your actions by Ed and Rich Eli and conversational fulfillment. Thank you, India. And thank you. This is amazing to see this
in like happening in front of us. And I want to give a big shout-out to multiple teams multiple product names that work together on this feature. If you imagine you have Android as a developer ecosystem actions, and Google is never easy. When you try to make both of these ecosystems work together as seamlessly as you're just being here. 3 cup as a developer, you could participate in actions by registering your act registering it up and in the actions in Europe in the actions rice email file. You can provide fulfillment with you or else either you or I can
play or structured data. You can avoid action usage using Firebase app indexing to help you like better. However, I've actions as we think of it is just the first step in your journey Productions. The movies with which you can enhance actions and further increase engagement with users. First you can use slices to enhance action to put a representation to reach you. I templates and second you can Bill conversation experiences and multi-modality experiences for Google Assistant. Take a quick look. places that has action for presentation to
templates like notifications or Android Google platforms, like assistant and search Easter Blitz support flexible layout interactive controls like sliders and toggles and dynamic like real real Time pricing so directly from the app on the user's phone. Places are compatible with devices on API 19 or above. Besides me the neighbors you to get get over 95% of our devices are out there pretty cool. This year's license to launch on Google search on Android. It's
tomorrow at 8:30 on stage for hope to see you. See you there. But what about all these new devices and platforms that are coming to Market things that you heard from Brad and develop a keynote this afternoon. You can extend your services to new devices and form factors by building conversation actions and multi-modality experiences for Google Assistant. Again food on more please set an article session on Thursday morning at 10:30 at 10:30 on stage for 9:30. I hope you are as excited about this project
as we are. Many Android developers are already building support for actions and hope you will as well. If you're interested, please let us know at G. Co attractions you can learn more about the project on the page sign up for further updates. Newsletters will be starting An Early Access program next month. Hopefully some other sign up for. I would love to hear your feedback and very specifically on built-in intense that they're building for intense. Or the next two days you have quite a few venues wanted anybody questions that you have their presence in office hours.
We are present in both 10 boxes in Android as well as assistant where we can show you more of these things live around there any questions that you have again, we have two more sessions one slice of tomorrow morning and one on conversation actions on assistant day after tomorrow and fairly soon. You'll be able to upload the actions or XML file using console. Thank you. Love to hear from you have any feedback give it on your doing this to us on this website? Thank you so much.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.