Justin Broughton has worked at Google for five years on building tools and infrastructure for developers. He is the tech lead and manager for Firebase Test Lab - one of the suite of tools under the Firebase umbrella. Prior to this he was VP Engineering at a number of large and small companies, working on building software in diverse areas such as Contact Centers and Health-Gaming.View the profile
Richard is a software engineer with the Google Play team in London. He works on the pre-launch report, creating tools for Android developers to find problems with their apps before releasing them. Before Google, he had a shady past in nuclear power, travel software, and mobile telecommunications cell planning. When not working, he loves cooking all sorts of food, but particularly barbeque. He holds a BSc and PhD from Cardiff University.View the profile
About the talk
Learn how to use the intelligent, autonomous app crawler to test your APK before releasing it to production. Run QA, set up test channels, and let the crawler test your app or game for you on a farm of real and virtual devices. Tests can be autonomous or customized: learn about both, and about new features in pre-launch report.
My name is Richard. I work in London with the Google Play console see and I'm here today to talk to you about if you don't report thank you the Braves to you've made it at this rather ungodly hour. The night off to the policy and I'm sure that that scary rather would eat Eiffel didn't help that. We really use the word autonomous marketing insisted you here for a waymo talk you're in the wrong place. Today we're here to talk to you about intelligence app and game testing that requires mercifully little work on your part and debatable and both
the play console and in Firebase test lab both products have massively increase their offering and we're here to tell you about what's in one what's in the other and walked in boat. I will tell you a story in the context of an app. Testing is a pain. What is it? You probably heard multiple times this week. It's really important. So as much as possible, we want to take Google's expertise and resources and make it our pain. I'm not yours. So that's not my talking about the pill Interpol. This is an automated scan of your app that aims to give you an early warning of quality problems
before uses see them. It's a free service. I mean applies to every APK publish to a place that will be two tracks bows. You may have had this week. Then you open and close testing tracks. We take apks that you passed a Google Play and we install them on to a bike 10 devices these devices find a range there a high-end devices like the pixel 2 and I'm too low and devices like the Samsung Galaxy J1 Ace. Once we have the APK installed on the device. We stopped a special Robo crawler process. the robo crawler explores your amp and putting Taps
and swipes and text as a user would This takes place on physical infrastructure, which is positive a larger form with thousands of different devices and a large diversity of hard way and operating systems. Your fellow developers have told us that they take pre-launch report very seriously in a recent survey 86% said that they consider pre don't report results to be important or very important when considering whether it's time to launch a new app into production. When developers have ignored a recommendation and launched something that we find
problems that about 75% of the time the problems that we find them. Ontraport would like to swing by real uses. Those uses got a pole experience. They may have left bad reviews. They may have left V right things. They may have chose not to spend money in the app, and they may even have any installed the app. Federal penal support test results I used internally within play as one of a number of factors when deciding how to feature on blank apps in the Play Store. As of today, we pulled over a million different apps on phone already crawling yours would really like to be
doing that. Now as you probably guessed why I said they don't suppose he's pretty good at finding crashes when we find a crash in your amp while we test it would give you a lot of information stack Trace is low cats things like that everything you need to debug the crashing fix it, but we don't just do crashes. We also take screenshots and capture a video of the Kroll on each device so that you can look for you at tissues in different languages in October. We had it the foreman stator. This is Network memory and CPU metrics so that you can look for performance issues in
your app. That's what my team does the playing field trip or what about your team, Justin? So my team works on 5S test lab so they test lab is a separate solution. That is part of the Suites of products under the 5/8 umbrella and uses the same technology as a free launch report including the larger device form the Richard mention it offers many of the same features. But you also integrates with a number of different tools such as Android Studio Fastlane E cloud SDK,
and of course the Firebase console allowing developers to test their apps and games at all stages of the application lifecycle across a wide array of devices giving you more comprehensive test results. So let's talk about now. This is Fitness. Now all Fitness but he's not a real application. It does illustrate real world problem problems. We know you will have Fitness pot is a Social Fitness tracking application. built by let's say a small company XP Solutions Fitness part allows you to log in with a Google account
or a custom accounts create an online fitness Avatar. Invite Fitness buddies with whom to train and even join a local gym. Richard and I are going to take you through how FP Solutions makes use of the many new tools at their disposal to develop fast and release soon to be wildly successful application. Thanks, Justin now is my wife likes to remind me when I'm being smug about something. My body shape is best described as a potato on the legs. So I really need
Fitness but to be very very good. I need all the help I can get. Pics of how many of you have spouses like Darth how many of you have way more time than you know what to do with every loss problem in your app. Nobody with all of us signals that pay don't report makes available to you. We know it can be hard to prioritize to decide which of these issues is the most urgent that you should fix the starting this week. We've launched a new page which digests all of the issues we find into a single launch
recommendation, as you can see here this shows up at the top of my new overview tab. They all Love You tab name displays all of the issues that we don't report his famed prioritized by severity. This new tab makes it easy for Fitness by the Sea at a glance. If this new APK that they have is a release candidate is ready to be released to uses or not. Now test Labs intelligent crawler does a great job of crawling the Android framework part of Fitness, bud. But the sport uses opengl for a an
online fitness Avatar creation part. previously the pre-launch report couldn't test apps and games which use opengl, but now it can If you are a game developer, you can write what are called demo loops. Demo Loops are an automated journey through your app. They can focus on a particular level or a particular game character. Damn, it looks are a great way. Fate they are an intense filter. 36th annual manifest they don't normally calls as part of your normal running of your application, but I'll test infrastructure will detect them and run.
Demo Loops launched last year and saw it. So hard bass test lab and starting this week as well. When we take your app to pay loan support rather than the usual Robo Corolla. We will run the demo Loop and then I will put the full sensor for you don't suppose we look for crashes and most importantly will give you those performance traces so you can use them for accurate benchmarking on a range of devices. If you don't use a demo do we will use another new feature called monkey actions when we detect that the Corolla
has discovered in opengl part of your aunt will full back to random input so we get at least some crawling for you. Fitness but really cares about keeping its uses data safe and secure. So Fitness bed requires custom log in or login with Google if you use 5/8 off both of these supported as I was saying before I was so rudely interrupted by that message from our sponsors. They don't report uses an auto log-in feature when we see a sign in with Google button, we will use it and we hold to magically have a Google account on
the device ready to be signed in. Applications that don't use login with Google can choose to configure pre-launch Report with a username or an email address and a password that we will again used to get past the login screen when we see it. next bus token ID place different sir way for years has to invoke functionality within you lamp. The bypass is the normal entry flow. For example, I use in my styles of searching the web but should be showing no search results within your app. So how many of you have deep links in your apps?
Wow. Yeah, how many of you test those deep links that every release of the app? Nice interview problem. They use deep links to encourage users to sign up to local gyms. The years are receives a 5/8 notification alerting them to a special offer. They click the link in the notification and then take him straight to Firebase to the gym sign ups. This is a very important flow for them because money changes hands launching this week. Please don't report when I test up to three deep links. You
can consider it an ad the Deep links we crawl as usual for the 5 minutes then at the end of the crawl. We close the app and visit each deep Lincolnton as we visit each deeplink. We prefer the 30 seconds and you can make sure that uses who follow deep links can have the same great experience. So as we have seen from the million-plus apps that the cruel as tested sometimes crucial parts of an application or only reached through some complicated flow like Fitness buds gym sign up form. Its
correct text bypass this phone. Swiss is Monroeville scripts come in. Is there another new feature to get past a complicated part of the end? They take the form of a short action recorded as a Json file. You can record rubber strips using a tool built into Android Studio with the amp running locally you record a series of time since wives. You got up load this scripts to preload triple X test lab. The Rogue one script guys the early part of the cruel carrying out the pre-recorded sequences to get the app into some state that you think will be interesting to test.
Once it's completed the robot script it stopped rolling normally. Not rubber scrub inordinately useful for crawling forms. They can also be used to guys the crawler post any parts that it finds tricky to navigate. Usability and accessibility really important to your amp if you make it touch talk is too small you might have a result like this. Not so no one of your guys at the gym Justin cuz I haven't even log actually know but I do seem to have it all the persistent irrational fear of treadmills,
and I've been on one since I think that's probably wise in the next few weeks. We going to add to pay loans for poor and intelligent kholat that's going to help think of a usability issues like not telling you about in the pool. Justin was trying so hard to click the Ozarks Asado small touch song. It's her problem for a wide range of uses including uses using your app on the go one handed or uses with dexterity in payments. Lunchable text is also excuse extremely important. So we will want you and text is too small, but we won't even the contrast is too low. Some visually impaired users
rely heavily on TalkBack. So we will also tell you when to voiceover annotations. I'm missing or confusing. As you can tell that Infamous button, I was trying so hard to click is included in the usability remote results and it's got a big red box around it. And thankfully as far as I'm concerned, it's Mark this should fixed. Softer. End of bug fixing Fitness by that loads of fresh APK to the Google Play console and then take another look at the Piedmont report overview tab, as you can see here.
There are no longer any serious issues to make a note versus APK into their production Channel confident that they uses will be receiving a good experience. So we hope that you two can adults the pre-launch reports as possible your decision to Launch. So that's where you guys can pre don't recall which is available for free to everyone who publishes apks to what we used to call the elf Ralphie to channels but no cold open and close testing tracks. You can find it in Google Play console on demand releases. Thank you, Richard.
So as developers, we know that the earlier we catch problem the easier they are to fix. So if you're a conscientious developer or you just like saving money. You want to catch problems as quickly as possible. This is where 5/8 test lab comes in. Now many of us myself included have discussed Ad nauseam about the importance. a creating automated test for every part of your application and to run those tests continuously to prevent regression. This is honestly the most important thing you can do to improve the quality of your application.
What writing test is hard? That's why when I Alston Gatherings like this how many of you Able to make it past the cover every part of your application way too few of you put up your hands. So I gave up nagging a while ago and we have Google I put a lot of focus into creating write greeting test for apps and games without actually having to write those tests. Earlier, we saw the automatic crawler. We did Spotify bass test lab as what we call a robo test. But we can do much more. Let's look at how we can create
test for all the different parts of fitness. So one of the first things that I as a user of Fitness bad will going to do its create my fitness Avatar which involves uploading my photo and from that photo actually creates a caricature of me. Which seems to have emphasized on my wood floor cloth? Is that allows me to choose a focus activity weight training? Is that really you? What appears that my ears and my vanity is around the various gym activities with all my equally
unrealistic Fitness Buddies. No, it's not what's our stuff is written in opengl. For a more immersive and fun experience for the user and previously it's been hard to test apps and games which use opengl but over the last year with Google have introduced a couple of ways in which you can now do this for easily. So we introduced to dial last year the concept of demo Loop suggestion and I discussed the idea that runs down the loops in your apps and games if you
choose to include them Now Dental loupes Rowlett Avatar stuff by simply recording and replaying Europa jail to Mom's and then during testing. They can verify that they don't speak crashes or performance issues. Now Fitness podcast run is demo Loops locally. On that device is using an app that we created this freely available. But more importantly they can run those demo Loops in farthest test lab across devices. Secondly, we announced at the games developer conference earlier on this year a collaboration with net ease to
create tests for games that run in 5 is Tesla. Netease is one of the largest games development companies in China. And as part of this collaboration, they open sourced that test environment test. Along with a couple of Frameworks that they use internally for developing tests for their own games. Now you can download this testing framework. There is a link at the end and with it you can create tests that add actions and assertion that refer to game elements. Let's see how that works. The Hitman test
And I'm just going to select game elements and add an action to touch the game elements. I can wait for a game element to exist or I can search that is visible that simple. like demo loops These tests can be run locally, but they can also be run inside a test lab across a wide array of devices. Who is the Gold Creations screen the fitness. Now my fitness goals are pretty modest. If you saw earlier, it's a good day if I don't fall off a treadmill. with this screen use this get to create. Fitness goals and share
them with all that Fitness bodies. now obviously Fitness pod Wants to create tests that cover this part of the app as well. not typically this involves laborious Lee writing tests in one of the many testing Frameworks such as espresso or you automator and then keeping those tests updated as the UI evolved because you made some small change in your app. Write an extremely painful Ryan sometimes me to I know exactly that is this is a really painful situation. What is the exact reason why we built
espresso test recorder espresso test recorder is positive Android studio and with it. You can quickly record espresso test by simply interacting with your device running your app locally. With it, we can create lots of tests that cover them know on opengl parts of Fitness Bud. Now, you can create these tests and either job, but the best thing is but if your UI does change You can go back in and easily record those tests. Let's see how this works. All right, this is Android Studio here. I'm going to choose to record Nespresso test. I'm going to pick
my local pixel 2 which is running. Then compiled and installs my app and brings up this event window. And this point I get to answer my name click on an app and it record the events old are in that window. Next I want to add an assertion so adding an assertion. involved pulling a screenshots from the device so that I can select elements in my screenshots on which I want to assert. I will do that like on an element. And then I got to click. Okay, I get to choose Java or Coughlin. I'm going to choose
Coughlin. I'm going to click okay? With espresso test recorder, you can multi variety or testing you can create test Snippets. For the commonly used functions in all your tests such as logging in and so with all that free time is excuses for not going to the gym operate. So now Fitness has a battery of tests and all they have to do is continuous integration system. Every time they make a change. They can be much more confidence. This is the beauty that he did not break the app in some obscure away is so honestly
priceless. But Fitness bad because I have to run on the Myriad of devices. The people dragged to the gym. How do they make sure that the app runs well on all those devices? Well, they could buy a whole bunch of devices hook them up to a workstation and stick him in a closet somewhere. But Fitness but wanted to pull up gym bags because they practice what they preach + maintaining a device closet is a really hard and activity. Trust me on this. Not only do you have to keep the devices online and updated but you also have to procure a new
ones at the old ones die or explode and fresh ones come out onto the market. Fitness pod would much rather be cutting code or pumping iron. So what's the alternative suggestion? I've already described to you how they don't support office freecrawler testing devices to every APK that glows but you can go much further with the fire bass test lab using a broader range of devices covering uneven Juanita gamma of screen sizes Android API versions different oem's and dip and chips at I'm
so to illustrate this we recently added pre-release P devices that you can now Old Testament, but what I'm most excited today to announce wait for it is on you support iOS so you can sign up for a beta through the 5/8 console. There's a link at the end. MyFitnessPal is a global application and as such. It needs to be tested on a whole bunch of different languages. Both 5S Tesla band play console allow you to do just that with 5 days with Friday's test lab. You can run your
tests on any look all you wish. And Fremont report recommends the most popular languages among Joy uses. So now There should be no reason for Fitness, but the waste all that time and energy maintaining a device closets, which gives me more time to spend at the gym, The one that they get me coating Donuts. What is the pre-launch reports v a test lab also give you comprehensive test results like crashes a day and I video screenshot logs performance metrics and map of the activities the crawler found.
The video and screenshots are especially important to help Fitness Fudd verify that they wrap looks good and runs. Well across all the screen sizes. and orientation and Lacoste Let's look at how easy it is to quickly run a test. in the Phi base console firstly I want to quickly point out on your way west imports. We going to run. A rowboat a troubled past is the simplest of tasks that you give us your app. Now I'm going to quickly choose a random from my hard disk there. And
upload it. While it's uploading you on that we can also upload robot script what you're going to do here, but now I get to choose my devices from amongst the many devices. I'm going to start with the default to devices, but I do want to add some look calls. I need a ride sign in. I'm going to add Arabic when I get there. And then lastly I'm going to add. Spanish and then I'm going to fix the college cuz I really only care about portrait mode. And then it's going to call the police that we got. Then I'm going to start a test a test because it's two devices times for Lookout.
So now what test lab is going to do is in the background it will allocate. Eight devices it will configure those devices setting the lock all the orientation. It will then install. my app on all those devices and then it will run a 5-minute Robo test. which will take five minutes, so The first of these you see we have logs we have screenshots. We have an activity map which is the map of the activities. We visited you have performance information, which is tied to the video so you can see what's happening in Europe at the time of the CPU spikes or the 8 frames per second
draw. You can find instructions on the web on the 5S documentation site, but how to hook test lab Optum any common continuous integration system like Jenkins full circle, CI 14 City. We have a number of customers who are doing exactly this testing their app multiple times a day. Haslab is helping them. Find problems way before they end up affecting that uses. And best of all you can get started for free. With all Spock feelings here. So it's really that simple. Justin and I hope that's okay today. Can you
use a renewed inspiration field testing strategy? It doesn't have to be difficult to text comprehensively. Let the pre pre don't report in the Google Play console help you discover crashes performance issues security vulnerabilities and soon accessibility warnings with zero configuration required from you. Just publishing APK today open or closed testing tracks and a report will be generated for you. You can go further than the older generation report by giving us custom login credentials. Robo scripts deep links on demo Loops to help us test urine better.
And if you want to test maybe more often maybe more comprehensively it take a look at which conditions of the features Richard just mention also allows you to create and run your own tests. But take a picture of this if you wish you a couple of seconds do that. So, I think those people already done that. We really love to hear more from you. So if you have any comments or suggestions, please use the link in the aisle app will come find us in person after the talk. Thank you very much. Thanks.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.