Mike is an engineering manager on the Google Play Console. His team are responsible for tools and infrastructure to perform safe, controlled releases to the worlds largest computing platform. In his time at Google Mike has also worked in Ads, Commerce, Google Cloud and on Google's internal build & release infrastructure.View the profile
Nicholas is a software engineer on Google Play Console. Previously he has worked on Wear OS, Cloud Platform and AdSense since joining Google in 2009. He has a bachelor's degree in Mathematics and Computing Science from Glasgow University.View the profile
Radha Narayan joined Google as a software engineer in 2005 from Cornell University, where she received a B.S. in Computer Science and a B.A. in Philosophy. She has been involved in several projects including the redesign of Google Maps, the infrastructure behind Google Search, and now works on improving the health of Android devices. She has moved across a few roles including program management and people management. She is one of the people behind Google's first Pride easter eggs, and runs a scholarship atView the profile
About the talk
Learn about the latest changes to release tools in the Google Play Console enabling fast and safe release rollouts. The session will also highlight developer best practices about release processes, including rapid iteration, fast feedback, and advanced staged rollouts.
Hi, welcome find Mike an engineering manager on the Google Play console and I'm very happy to be here today to talk to you about a subject this particular importance to my team and I am not is enabling success on Android through space controlled effective at Felice's. We know from yourselves from our own experience and experience about the teams are Google that releasing new apps and updates. It's what you might describe as an exciting time. It's the time when all that hard work, the whole team has been putting in finally gets into the hands of your users
and that is an incredible feeling. Uses a going to be delighted by your new functionality and improve performance. Business metrics I got to go through the roof and everyone is going to be catching on to that good kind of excitement. Innocence for a good release. We need to Delight uses with new experiences. We need to maintain and improve those important business metrics. And of course we need to get that text right to deliver a stable high-quality product. simple right But Melissa's
bill super scientists with some challenges. We need to keep our existing uses happy. We need to make sure we're actually improving those key performance indicators and of course not introduced performance or questions or bugs. If we do get this wrong, they can lead to an altogether different kind of excitement. I miscalculated release May result in crashes and bad user experience and this can have a pump a measurable impact. Recurrence the numbers on the subject and not dating shows. Uses a
26% more likely to give one star reviews on the days when the app is that is precious. Four games this increases recycling 52% one star ratings. Simile in reviews for reviews given on this precious days uses a 58% more likely to mention stability and bugs. And as you might expect this is not just limited to ratings in the store Key performance indicators. I've also affected engagement metrics uninstall numbers also suffer when you are discussing. On top of all this let's take a moment to think about the ecosystem in which your operating here, but more than 2 billion active devices Android is in
fact the largest software and Computing platform on the planet. Of course, this wasn't phenomenal opportunity for scale and reach to influence an incredibly large. However, it was 3% 2 minute challenges when it comes to software deployment. Incredible diversity and uses also comes with unprecedented diversity in device types with thousands of different makes and models across a variety of different form factors. And so Stop great releases at the cake. Was that good kind of excitement.
We need to listen and validate functionality improvements by collecting feedback from a representative set of engaged enthusiastic uses. We need to measure to make sure we're having the right impact on those key user and business metrics, and we need to optimize that technology by detecting those folks instability issues. Not only this but ideally we want to do all this as soon as possible the sooner we catch problems the best of bees and bugs usability issues or business metric progressions. The sooner we catch problems the most the most capable of purses are
the most active atoms can be on the faster we can ship even more great features. And this is where the right tools I'm release process comes in. For the next part of this section we walk you through the sweet to so the Google Play console makes available to help you handle this process confidently and effectively. We'll look at the plane. You're released to increasingly larger audiences and what you can hope to achieve each phase to reach the ultimate goal after playing a great new Apple update to everybody. And after that we'll welcome bother from the Android
system Health Team to give us a deep dive on the Google app for this process will see how teams at Google big and small. I'm making great use of these tools to scale that released purses. I'm so to begin all that. I'd like to welcome on stage search an engineer on the play console to talk to us about the first phase of a release testing. Thanks Mike. First I'd like to do a show of hands who amongst you have published and not in the play console already. Wow,
okay, and please keep your hands up. If you also have been using the offer and beat attract for your app. All right, how are you guys are great. I think you love this talk. Of course for those who haven't yet published in the app in the play console a track is a way to define a group of users to whom you can publish your app. For example, there's a production truck which targets all the over 2 billion Android users. And then there is the alpha and betta track Wichita get too much smaller subset of uses. Of course before you publish it after the world testing your
app with that small group of users. It's really important. Horror, this is for catching bugs issues. But also to scale and move faster feature. This is just to meeting your test version of your ass to a small group of users. This could be either your y te team product managers are QA team or whoever can provide you with early feedback. There's lots of ways to do that one way would be to Simply email apks to your users. That's quite cumbersome and actually had put the burden on their shoulders. While the alpha and betta track can
be used for Distributing custody case we heard from you that actually the Alpine get your chocolate to slow for that kind of fast testing. This is because that will see you later. I'll find it attract a design for scale testing with potentially millions of users and the atk's published those tracks go to processing that can take several hours. Turntable fast nutritive petting reintroduce. The internal test track is a new track in the play console that is
designed to distribute s versions of your app for small group of users within your organization. Let me show you how it works. switch on the computer I have your house my test out that they created for this demo call Il demo and I have it here in the play console. This is the religious page and we can see I have the production truck that beat attract the alpha track and the new internal test track. You can see here. I have version actually, I have written that 49 already published in a release in this track.
Griffin 49 here. I have one sister for this app switching to the device. And you can see this and you see on the device. Very very soon. Yes, so this is my device and you can see how this in the launcher. When I open it, there's a single activity and it shows the version code. I'm guessing that is a bit more sophisticated. If I go in the play console. I can see that I don't have any updates. So the version code of my APK on my test. This device is the same one that isn't the release that I
rolled out in the internal test track know Mike here just told me that actually my app doesn't conform with the official Google IO branding for the color scheme. So I will go and fix this and quickly distribute an app to my sister going back to the computer. So we'll go to Android Studio here and I'm just you know, I have my projects already open with my app. I also have installed a great old play pug publisher plug in which is set up so that it automatically publish the play console updates and Deals I make up my app. I have a
griddle Top Gear Republic APK release and when I invoke it automatically pushes, do, you know test track and building my app so I will create a new version version 50 And then I won't change my color to what was it again, right space pineapple. All right. So what will happen is that when I vote this riddle task? It will automatically. Build APK and then upload it to the play console to the internal test track. Once the upload complete. I will start a timer here. I have another device with the timer and a timer to 2 hours. I hope you guys are nothing to hurry. This is
because this is how long you should expect it to take if I was to publish this APK to the author do to attract and then we'll go to the Play Store on my testers device and it will check for an update. And as soon as the update is available, I will stop the timer and we'll see how long it takes. All right. So I started to ask so let's switch to the device list to understand you is building the ATK. And I started to upload it. Of course. I don't usually do this and only Wi-Fi network in a tank shared by two thousand people, but hopefully still
works. All right. So now the upload has completed device or I start a timer right? I check for an update. Actually that took less than 5 Seconds. If I update my app and install the update open it and I have in my new color scheme with my new version code. This is dramatically faster than publishing to the outside beat attract as you can see, but you guys deserve it. So which seen how you can publish the play console using a greater plug plug in an Android Studio, but of course, you can also use the
play console UI I can do just that. You can shut off testers. All right, so you can set up that stirs by specifying a list of email addresses the testers often with a single URL. Then the testers use the Play Store app on the device to install and update your app. It's super simple for testers to set up it works on any Android device with the Play Store and there's no special app or permission. It's fast safe and simple. Sorry, my just two buttons here and 50% chance of missing is all right. In order to get apks out quickly with this
Museum to testers before the processing of the APK has completed the application bundles published today in Toronto Stock are still subject to play policies. The update for internal testers is also different than from alpha beater. So users can be either Afrodita tattoos or internal tattoos of a hard to testers and the until test track / app. Genuine come with us. It supported by the play console publisher API in buy mini play console publisher plugins. So that means that you can
you can automate publishing to the intro test track very easily from your existing tool changed and workflows. You can get up and running with very little change. Ulta stock is great for Quicken to rationed during development for catching bugs early for getting fast feedback from your team and also for testing playing to Grayson because your app is published to play. It supports all the Play features LVL testing in app purchases and most importantly the new Android app bundle and dynamically features.
So if you're thinking about trying out those features the enchanted stock is a great way to do that. Once you have your app testing internally with intro test track you really for real user. Although typically you not get ready to open up to anyone. You might be looking for trusted feedback on maybe a confidential set of features allows you to stay while maintaining control of cool can catcher app has been supporting close testing for some time now, but we heard feedback
that as you have scales the outside eat attract. Don't feel well with various different scenarios what this example you would lie to test your app with the track for each feature Branch or what if you would like to create a new track for New Advance on Milestones, but I'm delighted to announce that now we can create new additional cost testing tracks enabling all those scenarios. You know how I have one production truck One open testing track any number of clothes. They think track and internal test drive.
When you want to create an additional cost a thing track, you can just click on the button credit course track. Choose any name you want and that time is also used in the UTI. From the early adopters resting that this is great for you two studies country-specific variants protein feature branches and testing and incrementally larger groups stopping a surface or how those contracts can be used. For those who have been using the author invita track you may have heard a penis report. It takes to
the play console and installed on 10 different unique devices and attached forms. It runs your app for 5 minutes at a time with Robo for nearly an hour of testing in total. It shows project performance issues and security vulnerabilities also added new ways to control the crawl like rumble strips and then we'll look for games announced any fighting you Futures go see they're awesome talk tomorrow at 9:30. Be sure to set an alarm for that or if you're like me
and you still wake up in a different time zone. As we still closed testing is great for what you want to test you out with a set of trusted testers, but I'll production uses of devices but then might not be as much as I hated as trusted testers. Am I engaged with a different set of features that you're up? When you want to have broad coverage of your code find technical issues or get more relevant business metrics, you can use open testing really allow any user to discover and
join your testing community directly from the Google Play store app on their device. And this user can provide you private feedback 50 Cambridge you in the play console and don't feedback won't affect your public ratings. This is a great way to scale off and to make sure that the new version of your provides a great experience to use this before you lunch at the world. If you need to you can also set a cap on the number of testers all of users that joined a testing program to keep things manageable. Of course open testing is only as valuable as users are willing to participate
today. There are over 230 million users who already opted in open casket with two and a half million users often each week. So it's really up to you to put it to good use. Late last year, we also introduced the new country targeting for testing trucks with country targeting testing. You can now control the country availability with each track individually. This can be used for both open and closed testing and allows you to run a test in a different set of countries to those in which reproduction app is life.
We have seen some of the teachers play provides to help you set your app. It's time to go to the next step and the release process and for the Arts. I'm going to hand it to Virginia Arnold engineering to play console who will talk to you about robots. Thanks Sarge. So you've tested internally then and closed groups and finally using open testing. You've polished the user experience ultimeyes those business metrics and ironed out those bugs and performance issues. You're confident this new release has it all the only thing left to
do is roll out to the public. Why style we unveil the new gated really slow for managing your lease? Thank you for all the feedback you provided based on this info. We further streamline the process making it faster to provide translated release notes and enhancing the APK upload process. So you can drag and drop all your apks NT release in a single step. We have also introduced support for instant apps instant games actions on Android and the new Android app bundle. In preparation for the new Android app bundle. We've
introduced the new UI to help you understand the contents of your APK or bundle. Even if you aren't using Android app bundle this provides useful insights into the size of your apks, the smallest possible means users can download and install faster and I'm more free space left for their photos and music. On average Absol see a 20% size reduction by using Android app bundle and the bundle Explorer can show you exactly the size saving you can achieve for Europe. We recommend all developers. You stayed
realize when Rollin Oats production stage for Lowe's I love you to control the percentage of users eligible to update and gradually increase this as your confidence and their release crows. And the unlikely event that something goes wrong. You can also hold the release. Well only a small percentage of users are affected. We're really happy to see the sense introducing the new gated release flow twice as many of you are making use of stage rollers. Following on from this we have Extended Stay True Lies with the new country targeted Roller Auction
for country and addition to buy percentage. This is really useful to start your lease in a market that you have a deep understanding of making it easier to understand your business metrics as the Rollo progresses. Once your lease is rolling eyes, you can use the release dashboard in the play console to monitor key metrics such as update rate and operating as well as crashes and and Ours. These metrics help you evaluate your release and ensure remains healthy as you proceed with the role. The statistics. You know, she was released of
ants making it even easier to understand the impact of a real life. For more detailed view you can use the vitals page to check my pics like battery usage tank weight clock usage and others. Where we've introduced many exciting your features for releasing your apps for the play console. We know that many of you all to make your apks appointment with continuous integration systems are using Gradle plug-in search that earlier. So we've updated to a publishing API and from today you can use these features and your automated work clothes.
Rolling out a release by the AP. I know uses the same release Moto you're familiar with from the play console to name your release and more easily modify release notes and weep relax the constrained somewhat. You can realize by the API to match the play console. The new testing tracks search mentioned earlier are of course also, no supported by the API. This is great for configuring your continuous integration system to push to your internal test track whenever you have a new boat ready for Q8. The new release Moto also unlock some features that were
previously only available by the play console. For instance. It's not possible to hold a release using the publishing API get to Philly Ultimate You released protest including automatically responding to problems. We've heard from any of you that well, you're comfortable rolling out test boats by the API you like the control of coming to the console when rolling out to production. So with the updated API, we added the ability to create and manage draft releases by the API. This means you can stay the draft release from your continuous integration system
and then have your product manager login check everything looks good and confirm and real life. You can find the food details of the new API changes and the associated client libraries on our developer pages and get hub. Turn the station off. I would like to welcome Radha to the stage to tell us about her the tools we have looked at our pit to use at Google. which way is hi everyone. My name is Rhonda and I run product operations for Android system Health. My team is responsible for making sure that all of Google's Android
developers can build excellent apps quickly. I'm here to talk to you about some of the lessons we've learned in working with Google's many app developer team how we use some of the tools you've heard about today and why we love them. Google has many Android apps you may have heard about like Gmail Maps YouTube or search would serve billions of users, but we actually have over a hundred and fifty apps many of which are just starting out and have a few hundred or a few thousand users.
Some of these apps have small developer team up to 10 people total While others have hundreds or even thousands of googlers working on them. In this room as well. There's probably a similar diversity in terms of how big teams are and how many users do you serve smaller apps are looking for ways to scale rapidly to a growing user base and larger apps are looking for ways to innovate safely and quickly. In working with Google's teams, we found that even this vast range of ass have a few things in common.
Each of these apps is constantly evolving is looking to add new and exciting features and adapt quickly to user feedback and they would all rather avoid that bad update and all of the implications that we heard about earlier. What I'll be walking through today is how we've come to a line on some common tools and best practices that scale from the smallest to the largest app. So if we go back to our key points on releases we need to make sure that we can listen to our
users and the feedback they give us we need to make sure we can maintain validate and improve the key user and business metrics and we need to optimize the technology to avoid bugs and performance issues and that support the incredible diversity of users and devices out there. How do we map this to Google and how do we scale and as teens grow in size than ambition? How do we continue to achieve good release attributes while still moving quickly. Well, we've developed some rules of the road
for how we can do testing an app updates using the tools we've heard about today. We've developed a few strategies to create a consistent process that scales even to the deployment of our largest ass. I'll be going over some of the strategies that Google app developers are taking advantage off to test out new features and to release the next versions of their ass First let's take a look at testing. We used to test by always putting the wrong most bleeding
edge features into our alphatrak this ensure that the team could stay in tune with each other and with the latest and greatest as we made progress. Unfortunately as the team's grew this led to an issue that we called you did what to my VP demo. Teams need to be able to move quickly to experiment and occasionally Break Stuff but product managers need fresh at most a day old stable bills to show off. The latest features do Executives and other stakeholders. We might need a stable
binaries. Let's say to do and IO demo that has a lot of the new features. With more testing tracks we're now able to give individual teams their own bills where they can experiment independently of each other and independently of the release process they can deploy daily or even hourly if they wish to share the latest features among themselves. We can also set the country availability of the testing tracks the global to allow Google in any country to test her ass, even if the
release product is not yet available globally. More trucks also allow us to have a more thorough testing process at release time. Thanks to individual team tracks new features have already been tested independently even before we cut our release. Once we're ready to release we promote are billed through multiple stages of testing starting with a team-wide test of tens of people 2000 of users and then a public open Beta with potentially millions of test users. Our open
betas allow external users to try our latest and greatest features if they're willing to give us private feedback on if they find they like it or if they find a ride a little bumpy. At first. We imagined that we might have a hard time getting users to sign up for this but users can join direct from the store listing in the Play Store app and some of our apps now have millions of users and they're open betas. What additional testing tracks at our disposal more and more Google apps are setting up an open Beta this quote shows how the Google
Search app is benefiting from the open beta program. It also highlights. That testing is not just about mitigating the risk of bugs and crashes, but it's also a chance to get feedback on features entry ensure that the new features have the right impact on key performance indicators. You might be wondering what we get from all of these various stages of testing. Why not? Just go straight to open Beta while we might want to test out some confidential new feature before they are visible to external users.
Why do we have both a team-wide test and a Google? Why test a Google we have a tradition of dog food in our own products, and we'd rather not break Gmail for all Google Earth AR test. So why do we still need the open Beta if we have so many googlers testing moments because even if a Google wide test can reach thousands of users by and large these users tend to have a somewhat narrow sets of android devices and open Beta gives as much more scale and a device set that is
much more representative of our production users. That's texting with validated her features and her app update. Let's move on to talk about how we actually roll out these updates and features to users. A staged rollout is critically important to Google team rather than updating 2 billion devices in one go. We always start small and ramp up incrementally keeping a close eye on business user and crash metric at the first sign of any issues. We could always halt a release. But how do we actually manage these releases
one method we tried for managing and a police was to wait until all the features scheduled for that release. We're ready then cut and roll. The release this work great at first but unfortunately features sometimes the sometimes hit a snag causing delays to all the other features schedule for the release. See you in the main release Becomes of the June release and now you're pulling more and more features into that single release more features needs more testing more bugs More release candidate more changes that make it hard to track down
exactly. What caused any particular bug or performance regression. And since the next release might be a while away developer scramble to get their features on board the release train. And then in the meantime you discover you need to ship an urgent bug fix that just can't wait and as you can see as the team scales this tends to spiral waiting for all features to be ready slows down Innovation and adds a lot of risk to the app release. We found that counter-intuitively
faster is safer. Keeping a continuous and predictable released rain makes each really smaller. Safer and easier to manage those urgent bug fixes can be sure to go out on time and interesting lie knowing that the next release is only a few weeks away developers tend to avoid cramming their almost but not quite finished features into a given release. So this is a great philosophy, but how does it work in practice? How do we keep the trains running on time?
And how do we ensure the new features? Don't go live before they're ready. To get teachers out safely in an app release. We use a technique called flag Garden feature development whenever possible we try to avoid shipping new features directly with an app update. We put each feature behind a flag that can be toggled on and off. from the server First we use the staged rollout to release the app update with all the flags turned off by first deploying such a change neutral binary. It's easier and safer to get this update to all 100% of our users. This also
allows us to evaluate the application binary itself for things like stability and system Health as it installs on the user's device separately from the feature and how the feature is doing. Once this change neutral app update is safe way to Floyd week and then slowly enable each feature by turning on the flags again carefully keeping an eye on all of those important use their business and crash metrics to detect any issues. If we find an issue always have the option to disable the feature by turning off the flag.
Importantly keeping these features behind flags and first deploying a change neutral binary means that any issue with one feature won't adversely affect all the other features in the app update. I could think of this as satellite deployment not that I've ever actually deploy to satellite. But step one is to get something into orbit. Once it safely in orbit. Then you can unpack each instrument one by one you can achieve this in your own app using Firebase remote config. The Google
Search app is one of many Google teams that have scale their release process and their future launches using flag guarded development. So you might be wondering how long releases take if we're going through each of these stages of testing and ramped up because the release drain stays predictable were usually able to deliver most app updates to users within a couple of weeks while having the ability to speed up a release if absolutely necessary. For the larger ice the ones that have more than a billion users were able to collect enough
data to have confidence in our releases within a couple of days. These release best practices help us keep each of our 150 plus apps that Google high-quality without ever putting the brakes on how fast we can innovate and bring fresh and exciting features to are billions of users worldwide. So that was a tour through the Google release process. I hope you found it useful and there's some ideas that you can put to use in your own or lease process and I'd like to welcome Mike back onstage
through posting stuff. Thanks, Rhonda. Took 2. Recap Racine. Why releasing is an exciting time and how we like to think about what constitutes a great release a release that the light shoes has improved business metrics, unstable and performing. After this vehicle accident 322. So the Google Play console makes available to help you achieve efficient. Well understood risk-free at Felice's we look to the new internal testing track for fast iterative testing with your team in testis. We looked at close testing when you the developer get to control exactly who can test your app Racine how
Google Play is going Beyond off of pizza with additional close testing to Monical's open testing to test your app from the Google Play Store Racine how this can bring huge scale to your testing. Wilson up to moaning out your releases with improvements to the release issued by improvements to staged rollout. And then you improve publishing API automation of your release using cradle and continuous integration tools. And which of the following tools to Google's approach to creating a consistent release Versa the scales from the smallest to the largest teams? We hope you find this
joke useful. Please do give these tools I go and let us know what you think. We look forward to hearing your feedback as you use these tools more and more if you'd like to share your experiences, or you have thoughts or ideas on how how we can make these two is even better. Please do come and see us in the Android and play sandbox in CC. Thank you.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.