Aylin is a Product Manager on the Actions on Google team and has worked at Google for 5 years. Prior to Google, she spent a few years in financial services as a software engineer. Aylin earned a Master's degree in Computer Science from Columbia University and a Bachelor's degree in Computer Engineering from Koc University, Turkey.View the profile
Nick is a developer programs engineer working on the Google Assistant, Smart Home, Assistant SDK, Android Things, and everything in-between. He graduated from Rowan University last year with a bachelor's degree in Electrical & Computer Engineering.View the profile
About the talk
Robust testing is essential for developing high-quality software and creating user satisfaction. This session will dive deep into developing end-to-end tests for your Actions, and cover the tools that are available to make the process easier. It will also share best practices on a variety of topics, like how to handle unexpected user queries.
Hello everyone. We are very excited to have you all here today. My name is Eileen and I'm a product manager for actions on Google near the Google assistant and iot Today we will talk about best practices for testing your actions and the tools we have bills to make it easier for you, but before getting into details, I want to mention why testing is so important. It's important because you're uses care about it. How do you know that? By looking at play the views we noticed 42% of the users mention if stability and bugs in
their one-star review. I'm 62% of the uses uninstall the app if they notice crash freeze or an error. On the flip side song 73% of the user mention speed design and usability in there 5-star review. Show me pictures of user wants Five Star apps and this has been the trend. We have been seeing in Mobile guards, and it's no different when it comes to actions. So today we will rock you to some tips and tools that will let you build your own 5staraction. Our journey today
starts with Jane. Jane is a node.js developer. She works going to start up and she wants to be with her own action. Her action is called today of Blazer and Trailblazer visitin nearby bike trails for the given user location. Together we'll start our journey designing and building Trailblazer with dialogue song. dumb people tested locally by using simulator I will show you how you can write your own unit test for your actions by using automated testing Library. I will show you how you can test your actions with your own uses by using
Alpha and Beta release environments. This is something we just announced yesterday and we are super excited to show you the details later today in the session. I want to deploy your actions of production will show you some tools flighthouse analytics. I'm stuck driver logging to make sure your actions remain healthy throughout the life cycle. Awesome. Let's start with design and build enables you to be rich conversational experiences. How does it do that? It gives you natural language processing out of the box. So it takes unstuck
to use a response and it automatically match that enzymes and anti TG Define and you can handle this in your own server. So it's a developer. You don't need to worry about knowing or writing your own machine learning models. It just works for you if you use dollar plum Let's see what it means for Trailblazer. So if you can see that looks like it's in between a system server and your own server so far today or Blazer videos not going to be around serverlogic. So I'm using said talk to player Blazer
assistant server take that response and takes are welcome in time from dialogflow and returns back to user. Softer that point using may say find me a nearby bike trails or where are the nearby bike trails and multiple other variations of this type of text responses and dialogflow automatically match that to instantly Define. Let's call it find players and that in time maybe reading the logic from our own server on 7 me may have the location services. Do you find a tray of anime response back to user and we don't
need to worry about natural language processing. Thank you. So now you build your action. How do you start testing it? Well, the first step is to use the actions on Google simulator. And this works on your laptop or any desktop and you tell us a test on your device if you have a home or a phone you can enter text or you can just talk to it. And you get the response that you expect and you do extra things like changing the users language and their language location. How does this work? Let's go to the demo.
All right, let's go to the demo. Click on a similar tab here who can jump right here into the action dress. I talked to Trailblazer. We can say find trails nearby I go there but I like flows going to be able to map it to the intent regardless. You can look if you have a request and response to have on the side and that will help you to bug your actions. Will say yes. I'm sure the location, you know, it's in the top right corner. I've actually mapped it to my hometown of Glassboro. So you can actually
show the different types of location to well. When did this place have you noticed that it has picked a park a park that is pretty close to where I used to live. And that's a very basic overview of how the simulator works for a second. So you can do this and manly test you act and every time you make a change, but as actions grow more complicated you want to make sure that users continue to have a reliable experience. And make this easier. We are introducing the automated testing Library factions on Google. So if you're
familiar with no to yes, then you'll be familiar with using assessing Library. It's written with no JS Sports all of the action on Google locales. And it works with your existing testing infrastructure, whether that's a lender or continuous integration, all those things worked at a box. So The three key features at this Library will do for Developers. First you can send text. To your action and us to be exactly what the user would say. This is important because you just can't say anything to your conversation anyway to make sure that you
can handle that. SML text responses and everything else that you would expect. And is your account to send the queries so things like use your storage and last time you visited the action? Will be exactly. But it would be for yourself. How's it going to work for you as a developer? Let's switch over the demo and I'll show you. Cos I got this test. JS file here and let's go down. So first we import B Library. I'm going to instantiate it when I passed in
my user credentials, so it's going to talk to this action as myself and we start a test and we get a Tesla name. We also set things like the location so we can change the location and then see how the action before I was in different areas and never will start returning a promise that we'll start the conversation with Trailblazer. Golden Garden send this query find trails nearby. And then we're going to send yes after it asks for a location. You make sure you also try now and test a different flows with different permissions. Very
angry to get back text to speech and cards and you can do a search in this to verify. Those are exactly what we'd expect the very end you and the test. How does his work? Let's go over here and run npm test. Okay, so now we started our test. And in a few seconds is going to send talk to Trailblazer. And when I get back response, okay, and you see a variety of parameters in that response is a Nike open text-to-speech SML cards Exedra. I found trails nearby going to get back that thing is Lucas going to ask for permission to use our location will say yes. Okay, and then at
the end you see you got the card you got the text and everything else from our action. You say you see that it says 1 test passes are there you go. So using this tool you will be able to more easily verify that your actions do exactly what you'd expect. float switch back to the slides awesome so far. We will Trailblazer and we tested it locally in our simulator and we wrote Our Own unit has and make sure they were all green. But it is always important as a developer to test your actions with your own uses. Why because they may
find box and usability issues that you didn't think are you didn't catch. Let's see how you can do it by utilizing our new Alpha and Beta release and Wyman's So awesome is the quickest way to test your actions. Once you submit offer it doesn't go through the review process so it can be live just in couple hours and you can invite your friends family and your teammates up to 20 tests and you can eat a late by releasing Alpha and acting on the feedback from your users and keep the
situation going on until you have the stable version. I want to have the stable version you can't tell me that 4 Beta release beta is life after going through regular Google review process so your actions would be like if there are fruit after the review monster fruit you don't need to go through additional review to deploy those to production so I ran to deploy this to public audience at the set your own flexible timing Can you tell me why it's wider audience for your bathing list after 200 uses? Awesome. Let's see how that works. In action.
Let's go back to demo mode. Perfect. So let's take it from where we left off in simulator. So now you can see in new release tab in the left-hand side navigation in the actions console. Let's go and click. Can you see stars from production? We also have now Alpha and Beta release. These are totally optional. You don't need to utilize those. But since you're in the session today, I'm sure you care about testing your actions with your own uses. I'm your father getting into the details on Alpha. I
just want to show you the versions. I already deployed before so canopy. I have three different versions once in Alpha Longston beta and wants in production. This is already deployed to my users. So let's go ahead. I'm Depot in offer together but before doing so let's change something in our actions so we can tell our users if they like the new change. So now you see the actions under the bill section. I'll go ahead and click that. How do you spell assembly with the Action Council you may notice that we just recently designed the whole URL for
you to make it easier for your navigation to come in time to hear that we already built with our flow. I'll click on that and it gives me option to edit that in The Outlaw. Let's see how that looks like. So this is my love you also have other in tanks that I mentioned to you before that we can handle in our server. so go back and go to default welcoming time right now are welcome message to the uses. Good day from Trailblazer. Let's change that and let's see if our Alpha
testers would like it. I'm still of Trailblazer EXT a good day from Google IO stage 7. So I'll say this. And I'll say to saved I can go ahead and see how that works by clicking right hand side. And this is going to bring my black version into the simulator to see the changes that I just did. So let's see talk to today old laser. And let's make sure it's reading from the draft. Let's try it again. Awesome. So it says good day from Google Iowa State 7 as we expected. So do you want to test it with our offer users and see if they're going to like
this response instead of the previous version? And also, please note that you can't see all your versions here and test them to see the difference. So I'll just I'll go ahead and click on the release stop now. I don't believe that. Let's put this version into our Alpha testers my production and beta remains like a different welcome that says welcome to Trailblazer are always going to be taking this version. I'm changing the valve, this phone's so we just need to click that. We tested it in simulator.
We gone through the checklist and we are complying its policy and wants me to submit this is going to wrong or please tell me checks and we'll be down in 5 seconds and what it is doing that or it's already done so you can see now That's all version. Beauty in deploying State perfect. So we created version 7 and now it's being in the deploying State and it will be deployed just in fewer cars and liked your testers around Texas UCI minutes off of tests on same applies to Beta by the way, the only difference if it is later, we would see the status under review
test has 20 section. You can add your own tests to make sure they are whitelisted to receive your offer up version so we can as Add this to my list and I already had some other emails before and once I hit save all these users are whitelisted to get my Alpha version just couple things to note here that uses your ending should have valid Google account is a Gmail or Google Apps user that you can remove and add a new users. The last step is a developer for
you is to invite your own test. So you seen here, there's an opening and this is static cling that directs users to your directory page. So if you're uses only using your action in their mobile, they can just search your action name like Trailblazer and they would see your invitation in the action director page. What is that are not familiar with that? You can just share this link. And once they click this link on their mobile phone either iOS or Android they would land into your directory. Like let's go back to
our slides. Perfect. So we know create an offer lease and we took 2 octoling share with our tester. So the first thing here is the place they London so this is my action directed a page on mobile and runs a stroll down. They see that the invitation that I edit them to my basal or same applies to all of them. And once they are monsters are all okay to be on Alpha tester. They say I'm in and after that point they will start talking to your test version. And at any time they can up. Or they can switch between Alpha and beta if you invited them to both
channels. How cool is that? This was something as it's this was this feature was a top developer request and we are so excited to actually and make it happen. So let's recap what we have been doing so far. So Jane build her Trailblazer action with dialogflow. She texted locally with simulator and also she was able to text via turn on device that was connected to her preview account. I see. How are unit test by using automated testing library and want everything was green.
She moved it to her teammates and made an offer release her teammates Gahanna feedback and she was able to ejaculate and make her actions more stable. And then she listens to her beta testers tomorrow wider audience and wants to make sure her actions are stable. She's ready to go for production. But before going to production I would like to share some best practices while you are texting your actions. We always think about answer and Johnny's and try to capture those use cases in all tests. But what does it even mean in a conversational space?
Showing up different world your uses maybe multiple mods. I'm Beast modes are determined by Imports and outputs of your device. Siphon user is starting to Trailblazer in Google home on the guest voice. What is the Outsourcing to Trailblazer on their mobile phone? Not only they would get voice but David also get image of the nearby bike trails. So I said it's important to think about where our users may be interacting with our actions and handle these test cases accordingly to
make sure their experience is smooth in all devices. That actions are offered. And the second important point is to make sure your users are able to answer your actions. So it's good to text your emo caitians in multiple ways, either explicitly by saying talk the Trailblazer SS Trailblazer or you may also have some beauty and intelligence or some implicit in my location that maybe tickets in different way lest. I play a game or find a nearby bike trails. so actions on Google today are offered in multiple languages and it's become
even more important to test your uses languages in the conversation about so this is a great song to be able to text programmatically by using unit test or in simulator as well as you can offer your offer and beta environments to the use of that speaks about off these languages and see if they can navigate I'm saying applies to location today Nick Point adult dating simulator. We are able to change the location and see how your action response back. Connections on
Google offers multiple functionalities to developers of payments account linking the engagement order update push notification. Some of these may be hot to test programmatically, for example, let's take payments as an example of perfect used to test with your alpha or beta testers and make sure that they can get the functionality that you want to be. I'm lost but not least our action may be touching into multiple stack. So Trailblazer rebuilt not only have dialogflow and assistant on the clients. I love you also be on our own server. So it's important
to make sure and handle all the cases in each of these texts in the architecture. All right. So so you've designed built and tested reaction now you get to users but it doesn't stop there. You have to continue to ensure that you just have a great experience. I have a couple of tools to help you do that. The first of which is the health analytics section in the console and this will show you graphs for the number of errors that users are experiencing as well as the latency and we do periodic checks on your action.
Unhealthy let you know over an email and if it continues to be unhealthy to be disabled until it becomes healthy again. What's 40 / 2 the demo? So here we are in the analects page you can jump over to health and use the errors that happening and let's go ahead and jump over the last few months. What kind of Errors have we had? All right, and you see that there are a couple of errors that start out a few weeks ago. We can go ahead and look at the errors and selves invalid response time out unknown error. I'm going to learn a little more about why use areas might be
happening. Also filter these out and say okay when people might be having a different experience on a phone compared to a Google home do like all these errors relate to mobile devices. Just look at the latency and you see for 50% of users the game about 200 seconds. And that's a pretty good response time as he jumped out towards the 95 percentile. It gets a little higher and so we went to see what kind of things are happening for some of those users to help reduce that latency. And is more grafts as well.
Second part making sure you have a reliable action is stackdriver and to show you how to do that. I'm going to go back to the simulator and Tackle about talked a little bit about finding bike trails, but it is also a bug interaction. So if I say, give me A tip taxi going to crash and that's intentional for this demo. Okay, so says final response must be set. You might have experienced this when you're using the simulator, but you might not know what this means view logs button directly into stackdriver, which is a error logging tool.
And I'll show all these things that happened all of the logs for every action. You can go ahead and Soldier something. Like errors, so fine all the errors that happened aren't you? See right now? We just got this fine response must be set. His life information here. But what exactly does this mean? Well, since we're using Cloud functions. We actually also log what's happening there as well. And we have this error in our class function. It says reference Hayward Field tips is not defined index. JS long and 30. Okay.
Well, let's try code and figure out what's happening. Okay, index. J s line 30s. I will this is actually correct okay so you might see that there is a syntax error in our index a JS file and once you figure out that this is a problem we go ahead and make the change and do another Firebase deploy and the next time you run air tool and do the simulator testing it will work as we expect let's jump back to the slides and recap what just happened So was that driver logging you can examine are logs for your
fulfillment and can filter these logs based on types of errors. And anything else that might be logging and it's really cool is that no custom code is required. If using actions on Google you get those logs and using Cloud functions you get those logs as well. Awesome. So this concludes our journey today. So, let's see what we have been doing together. So it started with Jane and Jane had an idea for an action called Trailblazer. She built it that with dialogflow.
And once she was done with designing and building she moved it to her simulator and tested locally on her device and on the Simulator 2 on the actions on Google web and then she rolls her own unit has by using automated testing Library. I want it was all good and test it. She did Leave It to Her Alpha and beta testers to make sure she gets at Sable action. And when she get that she was able to deploy to public all the systems uses. And she was able to grow her audience and she was able to get
more engagements and throughout this journey One Direction was in production. She was able to monitor it by looking at the health analytics in actions on Google console or stackdriver logging that comes out of the box and was able to check and monitor and catch the box if her production uses have been seeing any issues, so she was able to fix those and make sure her actions was housing. I'm is a result. Jane was able to get her own five star actions. How do you want to make sure that all the tools and tips
your share today with help you get your own five star actions? Thank you so much for coming and we would love to hear from you. So if you can go and try not to be back in this thing up here, we would be so happy. And also please visit actions. Google.com. If you want to learn more about the things we talked today and Nick and I will be available after the session in sandbox. If you have any questions and I would love to hear from you either through all developers channels or in person in eye on her and make sure you follow actions on Google
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.