Jumana is a product manager at Google. She works on the Firebase Predictions, a machine learning powered predictive service for mobile games and apps. Before joining Google she built mobile products at Zillow. Jumana loves helping developers grow their apps and build delightful experiences for their users. She enjoys traveling, scuba diving, hula hooping, and prickly machine learning problems. Jumana holds a master's degree from the University of Washington.View the profile
Todd Kerpelman is a developer advocate at Google, which means he primarily spends his time making YouTube videos and has somehow convinced Google to pay him for it. As of last year, he's been spending most of his time talking about Firebase. Todd has two small children, so his hobbies primarily consist of falling asleep on the couch, attending Moana-themed birthday parties, and reminiscing about the days when he wasn't too tired to have actual hobbies.View the profile
About the talk
Building a high-quality app that receives rave reviews and accolades is great, but it turns out that your coffee shop down the street doesn't accept "accolades" as payment. You also need to build a successful business. Firebase is here to help you! In this session you'll learn how features like Firebase Predictions, Analytics, and A/B testing can help you grow your app's business strategy from "cross your fingers and hope something works" to a comprehensive data-driven strategy for success.
Holy cow. It is really early be super excited to find out about how to make her out more successful with Firebase and I've been up for hours. Anyway for East Coast jet lag people Europeans use coasters internet by internet. We actually both kind of had to run here cuz we had to kind of get through the security line and get here in time. So you're going to relax and drink some water while I do the talking first as we all know building a great app is nice, but it is only half the battle. So yes, we all love
getting like nice review stores and you know, something's up and nice comments from the people that use But it turns out our landlord usually does not accept those as payment even in San Francisco. You still often need to make money. And so if you're like a lot of developers, maybe like a typical game developer, you've got a business model that consists of like selling some in-app purchases and it may be kind of crossing your fingers hoping for the best didn't and maybe that's it. If you're feeling ambitious, you might be doing a couple of tests to help improve that but you know, this
sometimes involves a fair amount of like one-off engineering work. This ends up being kind of hard to scale. Well lucky for you. And for this talk to this is one area where Firebase can help. Firebase does help you reduce the time. It takes to build your app in the first place with lots of great products like Cloud functions and caught fire store and cloud storage and other stuff that probably begins with Cloud cuz we like that but we also provide you with a bunch of out-of-the-box solution that can help you grow your user base retain it and monetize it. So let's talk about how we can
move arm on sedation model from like, let's just try some stuff and hope something works to something more sophisticated and data-driven so, There's nothing wrong with help but as my fellow product manager over here and we'll tell you but verify so we could make more money a lot of questions do this. What if we decide that was a bad idea do we have to relaunch the app? This was a bad idea. How do we measure success? How is all of this going to work? And so we can walk through this talk with how we can't how far away is really can help train her team achieve this and other ideas that might have
So first off, let's go to Hope but verify so far. Hope is verify the first thing you need to do is actually start looking at the data and looking was exactly happening inside. So that's where the Google analytics for Firebase SDK can come in to help even if you feel Totally jet-lagged and unmotivated without anything else no customization. Just single line of code integrated SDK gives you a ton of value insights about what's going on inside your app. For example. We capture automatically all in app purchases that happened
inside the app. There's also a lot of information about user engagement session time and so on retention retention cohorts and how long you returning users you're acquiring a certain weeks daily active users and other demographic information. He is such an important example in this report. You can see which of your in-app purchases are selling the best and how much you're making off of them and who is really buying them. And and all of this religion is SEK is is free and Reporting is Fiona and there's unlimited 500.
And if you're feeling a little bit more motivated what she should one line of code get you a lot more of what's really happening inside of your app. So you can say if you have a game when they went around is over or when a tutorial started. What is the total ended when somebody joined the league when somebody shared the app for the game with their friends or video? I need such important events in side your app that gives you a sense that the user is making it through the use of Journey you're intending for them inside your app one line of coke. And once you have that then
you can start asking a lot of questions specially if you can buy that with big query as well so you can ask questions. Like how easy is is level 5 or hardest level five and how many people are making it through the tutorial and so on. What you have those events and you can actually go ahead and from the Firebase console label a few of them as conversion about so important. It's a huge milestone in the user Journey with your app. So conversant events coming to place throughout the fireplace platform and you we will see how later on there were important to help you find a whole bunch of
targeting. For example, you can you find your ad campaign so that whatever ad platform you use that you're saying get me more users who are more like this or Horde likely to complete one of these conversion of abs So talk about this more later and say student. Well, okay now we can start changing some things and see if they're having any kind of effect and we're going to be using Firebase remote config as the primary driver of these changes before who has use remote config mostly clustered in this area. So I'm going to
talk to me before the room cuz y'all know what's going on. He Value store that lives in the cloud that your app can use to change like the look and feel and behavior of of how it works without your needing to publish a new version of your app to the app for the Play Store. So the way it generally works is you're going to set up a bunch of values locally on your device. And then as time goes on when you want to change one of these values you can use that using remote config, which we will do in the cloud either using the Firebase console or using the rest API that we just
announced like a month or two ago. If you haven't tried the rest API a lot of people are asking for it it's now available so you can make changes there. We got one fist pump. Thank you excited his bumper. So basically then when your user starts up there after next time remote config is going to download these values from the cloud and apply them on top of your default values. And so then when you ask remote config for the value associated with a certain key, you will either get the default value that you supplied originally or the new value that you supplied from the
cloud. I see the thought bubble over the audience that says what about my user data plan? Is that your only downloading the Deltas write the values that have changed since the default that you supply or you know, originally locally within your app and that means you can basically keep as many values as you want in remote config like hundreds and hundreds and hundreds of values. And because you're only downloading the values that you want to change from the cloud, you know, the actual amount of data that your user is downloading
is still nice and small I'm in it doesn't end up being a big hit on their data plan or battery, which is nice. Look at any value. You might want to change within your app, but that's like the look and feel of your buttons for the difficulty level of your game or you know how or if you're displaying ads or any of the way items are organized in your store, right everything that you've gotten like one of these constant file somewhere. We all have one of these somewhere at a few hours to refactor this and get it to go through remote config. Once you've done that you basically have the
power to change anything in your app directly from the Firebase. Console again without your needing to publish a new version of your app to add to any of the various stores. So for instance, if we've gone ahead and parameterised our store layout as a variable that we are running through remote config. This isn't something we can easily update through the Firebase console. Like we add this parameter publish it out to the world and that lets us easily Implement Jane's idea and and work
right now. It's up. Wait a minute. They want a glass of wine to go with it and also more yet and also more ways to monetize. So it turns out now that was added the ability to change her story that has given Jane a bunch of other ideas that that she wants to try. Well now she's thinking okay, it's great that we're changing our store layout to show the most expensive items first, but will that scare off like Two players are players that haven't made a purchase.
You know, what if we only change your store layout for players who have purchased stuff in the past. So maybe we can try to maximize our you know, average purchase per user while also encouraging our new users to go in and you know, try out try out something by just buying a little thing. So this is something we can actually do pretty easily with the systems. We already have in place. We don't really need to do anything new here cuz see when you implement analytics you have the ability to create audiences and these are users who made certain kind of arbitrary criteria within your app like
some combination of hitting certain events a number of times and I'm being thing aside a certain user property and one audience that analytics will create automatically for you when you started up is this purchasers audience, which is basically anybody who has ever spend any money in in your app. And one nice thing about remote config, they didn't tell you that earlier. Is that a contains a bunch of logic on the backend that lets you send it different values two different users depending on the situation so friend since you can send different values to your Canadian users, if you want to
provide like a nice localized experience for them or you know, you want to test some version of your app in Canada before rolling it out to the rest of the world. And another thing you can do with remote config is send different values two different users depending on their if they're in different analytics audiences like, you know, this purchase is audience. We just talked about to go I so we can tell a real can shake hey, you know what? Let's create a special condition for People based on whether or not they are in this purchasers audience. And then we can change our store layout only for
people that are within this group. I thought we get to implement James new monetization strategy. Where's my heart? So why the snakes are happy it also kind of get their thinking. Well, it's great. We real to change her behavior for users who have purchased stuff in the past, but would it be nice if we could also change your store layout for maybe people who we think are about to spend money sometime in the near future. Well, gosh, that sounds great, but nobody can predict the future. well product manager Firebase prediction
Austin mice under and maybe depending on how many maybe if we can get some help from a robot friends. So predicting the future as you may have guessed machine learning is a fixie Matt and I know this year and so is mindfulness and here's a product that tries to combine both directions last October and it's a batteries included predictive service. That means you don't need to have to do anyting know to take advantage of it. You just have to be using the
Firebase platform a locking your event and we will do the machine learning on your behalf. So basically what we're trying to do here is use the app Telemetry in order to discover user Behavior, so For example, if you're sending us all of your Google analytics through the 5 SEK, we can start looking at looking at some of the behavioral date of the user certificate. For example your past fenders. So we'll take the past fenders Wilbur the tensorflow model. If you don't know if there's a flow is Google's library for building and maintaining a machine learning models. So you
don't have to know anything about that before you will choose the parameters for Piper to knit on your behalf. And then we'll start learning about the patterns of behavior that exist in your app the first two behaviors. We learn our about who's going to pay and was going to stay so spend Turner's Spenders. We look at their past behavior, and then we can look at you use a group and then help you identify the group of user who is likely to spend in the future to spend time trying to monetize more of your this is the correct group of user to kind of look address
potentially with more promotions or something like that. Just hit it by them. Do a little bit more spend. So you may be asking yourself. What can I do with these groups? Once we created you can use it anywhere that I already showed us how to use an enema configuration. So you can use it in the mud configuration for targeting you can use it in notifications or you can use it in a b c a b experimentation. The basically the idea here is to find the right groups of users so that we're not bombarding everyone with everything all the time.
So just you taking more targeted actions and you're getting a lot more conversions in your app without kind of bombarding all your users with everything that's happening all the time. So how would you take advantage of this you might need several years of graduate study in machine learning? You can do it without that. It's a really worthy topic of study by. I mean it's about 2 weeks to clear the first predictions, but after that and a blessed every day, depending on your volume and run out while I but it's
French word for these predictions are updated based on actual user behavior on daily basis. Should you search Mako from your predicted to spend for this is not spend from being predicted to turn to rain predicted to not turn you can start asking all these questions about groups of users in your app. If you went ahead and did instruments your app and add his conversion events. Now we can start asking these questions about your Converse. Events and using the
predictions to help you do users go along the journey that you want them to have it at your app. So you can start asking who is Sherra Wright Ave or who will join the League or who will each level five whatever event you logged as a conversion events. Now we can start asking a prediction about it like who is likely within the next week say to actually participate in the direction and only sort of like show that share this with your friends to people who are likely to engage in that says push people who are already and behaviorally ready to take certain certain events inside your
app. And so talk already defined 32 help getting along with taking that start with A and now instead of having it show up only for the audience Spenders purchasers fetus from sisters now we can I give it to people who predicted to spend and then with others that condition and we are good to go and that makes Jin super happy. Look at that ecstatic face. So now she wants to try it a lot more stuff and she's worried. It's going to break this initial success. So maybe we can help out
to convince members of the team that her plan actually was a good idea like the trick is how do we know that all these changes we've made have actually worked but even if we saw and not taking spending while we were changing the store layout, you know, how do we know that wasn't due to like random chance or another promotion that we have to be running at the time or some Weather change that we made elsewhere within our app. I mean, how do we know? What if we hadn't made this change? We would have gotten even more Revenant right? It's something that that we don't really know and so
instead of relying on like punches or confirmation bias. We can prove it with data and that's what Firebase a b testing is for. So this is our tool to help you make sure that all the decisions you make in your product are backed up by actual data and not just kind of, you know, lucky guesses or things that you want to be true. So on the process of creating a t-test is you're basically going to Define an experiment and there are three steps involved whenever you want to create an experiment. The first thing you're going to do is going to define the group of users who you want to place
into this experiment right? Maybe it's like 5% of your population, or maybe it's your users who speak Korean or maybe after users who is Mark is like super competitive players and analytics and do you know you want to try some new multiplayer feature on them the second step in creating your av-test is telling a b testing how you're going to create different experiences for each of grants that you've put into this experiment speaking you're going to do this by delivering a different remote config values to each of these different variants. And I think it's important as we mentioned several
slides earlier. You can basically take any constant in your app and wire it up to go through remote config which basically means if you can Define it in a variable, you can probably a t test it and so you got a fair amount of flexibility with what you can do with your AP test if you wire up your app to go through remote config for all of its constant that you're setting so there's a lot of flexibility here. The final step is telling a b testing what your primary goal here is of your experiment. Right? Is it to improve attention? Is it to increase your in-app purchase revenue is it, you know
to increase the number of people who have made it through your onboarding flow essentially if you can measure it in analytics, you can Define it as a goal in your av-test and on this is useful not just for like, you know, helping to create your conclusions, but maybe testing will also use this to refine your experiment along the way to make sure that it's finding you exactly the variant that gets the exact maximum value and the thing you're interested in as your main goal of your experiment using some fancy Bayesian statistics that I don't really understand. And it basically you're going
to go to relax right to hang out with your friends in a hammock inside of giant glass Dome. Apparently they're going to run out of oxygen to be panicking lower original hypothesis with data using a Navy test in this is what it would look like going through the Firebase console. So the first step to finding who were going to put into our experiment. So we're going to stay at our target audience is going to be users who are predicted to spend money within our app and maybe we'll run our experiment on like 30% of those users.
Then we're going to define a whole bunch of different store layouts here by providing different values to the store layout key that we are running through remote config. So here we have, you know, three variants in addition to our control. I think we allow you up to seven variants plus the control group, you know, so it's not just a B test. It's a b c d e f test. And they were telling Firebase that we want our goal here to be maximizing are in-app purchase Revenue. I know that in addition to our primary goal. You can define a number of secondary goals and all of your experiments and
this is useful just for kind of keeping track of the overall health of your app as you're running one of these a b tests, like if we make a change that like really boosts our our Revenue, but it turns out we're so aggressive and pushing these, you know pushing the store that like a lot of our users kind of get frustrated and leave and stop using rap. We want to know that and so that's sort of the point of having these secondary goals and all you're a t test is it gives you a way of kind of Sandy checking your experiment and making sure your app is still over all healthy while you're in
pursuit of whatever your primary goal is. Anna basically check back in a couple weeks and see what happens. So this is sort of what a sample report might look like. We probably don't have time to go into full detail on all the charts and graphs that you get to be important. Take away. Is this sort of giant line of text up here in plain English that tells us basically out of all the options that we tried showing the most expensive items first was in fact the proper way to maximizer in-app purchase Revenue at least among our users who were predicted to spend money. And by the way, this is not a
completely hypothetical example in Digimon. Twist, you were playing along. We had our friends at rocket bike games actually do that. Same thing as she see them walking around that are really awesome. Say hello and give him a high-five that is small and actually in a b testing multiple times and a lot of these experiments and Abby and actually inconclusive they couldn't find out a good layout for the store that maximizes Revenue but then they try to change the layout of the beast on the predictions. I was skeptical but it's still
predicted to spend they will put in the game that are important on top and for users are predicted to not spend as you can see in the second one. They'll put lilies Crystal packages and a special one. Suck a high-value one for those who predicted to not spend more in the game. So the crystals are some sort of an in-game virtual currency. And in both situations, they increase their of invite about 25% for the predicted segments 25% increase and that was
actually really really great for them. So all these items are available for everyone. It's just a layout of the store was done differently. So it was a worthwhile experiment for them. And is totally medicated and now that encourages So This Is Us very good as you can tell it's trying to prove himself. It's okay. Grace is here and he's like, okay, we want to experiment with whole new ways to you no more money, and we're going to be open to new experiences and with a B, testing and the farmers platform. I think we can we can help that team
along with some of more experimentation self. Ellen wants to start showing ads now that she is cared except for a lot of apps because you don't want to compromise sometimes so user experience with ads that could be true savour at the wrong time or any of that. So that's why a b testing here. Henry to help us and Them predictions can also help us. So if you have before you can go ahead and install that secure from the Firebase console and the SDK has a lot of advantages one is that in the first place, so we can get all sorts of and Admiral, So I guess
it's about which placements of your ass is getting what kind of exposure what kind of clicks and which ones are generating money in that genetic money, but most importantly as we're talking about experimentation a b testing and that stuff is that we actually predefined goals for estimated revenue and total revenue. So that means when you're running a few tests to ask you can make sure as part of your goal that not only are getting admob Revenue maximized but you're also your total revenue maximized which means you're not cannibalizing your in-app purchases. You can't even sit all
three of them as gold so you can keep a closer eye on what's Happening throughout throughout your ass and all your monetization. hot streets shorter word it out. So free samples again back to our game example at the end of example is predicted to and not spend you will show her what a video or something like this so will go ahead as his experiments producers place is not to spend to get a shower and go to word. It has the goal here is to maximize our total revenue again. We don't want to maximize just as Revenue because we want to make sure that in
app purchases are not hurt. So that sounds great. It's a lot more optimistic now and he wants to try perhaps owing interstitials at some other point of the game showing its residual sounds a little scary, but we can do it with an AP test and see how it does. So for it not to spend less a way for you to find a remote configuration that controls how often these are being shown. We cannot show them at all show them more often or less often choose. Your fetus is not to spend but in our goal this time in addition to keeping an eye on the money, we keep our eye on
detention. So we asked for daily user engagement for retention to make sure that we're not compromising the overall user experience, which is something you really can't even focus on with Firebase. Yes. Guess these things here isn't just about making money. I know that's kind of the whole theme of the presentation but like your game design team for instance could use a b testing to make sure your game, you know, like fun turns out fun games usually make more money to that's that sometimes helpful. So, you know, we can, you
know, try like tweaking the difficulty of a game to see what our users actually like playing more so we can say hey, you know what, let's take 10% of the global audience and try the difficulty levels tried changing a few of the Variables answer Bar base game setup that makes it either harder or easier for some players and then see which one kind of increases our goal of daily user engagement write your players appreciate the challenge of a more complicated game or make them feel challenged. Do they get more involved or you know does increasing difficulty level drive them away. They get
frustrated they give up and actually they wanted an easier game where they win more often cuz that makes them feel better. Right one, who is a former. Game designer I would have loved to been able to run these tests. I often just kind of in the past had to guess and hope I got it right and spoiler alert very often. I did not so be able to run and stuff like this directly from the Firebase console would have been fantastic. So all of this is nice and helpful. We don't just want to kind of sit back and hope that our users spend money part of having an ongoing successful business is staying in
communication with all of your customers or users. It's a dialogue not just a monologue. Sometimes I considered beef or just a nice feel good moment, you know, make sure do users know that you care about them and you know, sometimes it's about driving players to perform a certain desired action, right? Maybe it's getting them back into your app if they haven't tried it in a while or you know, like trying out a new feature that you want that you think they're going to be excited about or you know, maybe making an in-app purchase. And that one way we can do this is through notifications,
which have very high visibility is as we all know, but they also do run the risk of annoying your user if you send them to frequently or to indiscriminately and then they go and turn notifications off which would be sad so you can have to make sure that you are sending the right message to the right people. So when Sarah from promotions comes by and tells us we're having a big IO sale this week. We should tell everybody all about it. You know, that sounds nice, but maybe instead of telling everybody we should make sure we're targeting the right people and we should probably make sure
that we're delivering the right message. So it had Firebase Cloud messaging available for a while to help with this and for a while. I think we've been saying hey one of the great things about Cloud messaging is you can send the same message to all of your users on Android or iOS devices with one call to the same service which is true and it's still nice and it's helpful, but I think would often gets overlooked is, messaging also let you take advantage of all of the targeting and experimentation features that we've been to. Hanging out today. So for example, if you want to announce a new
sail to your players, you could send it to all of them or you can send a notification only to those players who Firebase predictions things are about to spend money or more likely to spend money and you're at and if you're not sure which version of your messages going to be most effective. Well, guess what you can av-test it right with 5 ASAP testing. You can try a message on a small segment of our population say like, you know 5% of your English-speaking users cuz I notification is in English and you know, we'll try to different versions of a message see which one is most effective and
tell him he testing. Hey our primary goal here is to get them into our in app store. Delete start thinking about making a purchase maybe something like that and then we can see which version of her message was most effective in driving players towards that goal. Then once we determine the winner, we can then go and roll it out to the rest of our users who weren't included in the original experiment so you can kind of make sure that your marketing campaigns actually move the needle on the metrics that Care about before you roll them out. Why lie to everyone. It is super nice.
Sitting out here that everything was talking about here today, whether it's been with you for the duration so you can a little extra love yes or no are starting to Jeanette predictions to figure out who's going to hit certain conversion event that you care about yes or running a Cloud messaging campaign to deliver just the right message to just the right users and tell him about your big say I did all of this really can be run by anybody in your team. You give designers for product managers recruit Engineers from the Firebase console. So you only
have to take a picture of it. Your engineering team has only a one-time setup hit to set up. For example, the the Google analytics to dream with configurations the notifications at CK's but then you're gross team can be running experiments for a couple of months and a saying productive and trying to maximize your oven in your session said you're up for the rest of the engineering team is working on other things. So you can't have everybody on your team being super productive. Addition strategies can go from
a lot of verification and iteration and a lot of a b testing that leads you to the best outcome. It's you're not going to get to the best version of that's right and you get to go from this actual money and coins but you forget I'm sure somewhere in San Francisco. There's a landlord that is accepting Bitcoin guarantee. So if you want to find out more about some of the stuff we talked about today, there's a much more detailed and Cloud messaging which is
happening later on today at 4:30. And if you want to find out more about analytics and some of the new things that are happening this year with analytics. I think it is happening, right? After the Cloud messaging talk to get a afternoon full of Firebase see things and if you have questions and want to talk to us, you can meet us in the sandbox. That's the big dome thing that is not box-shaped and doesn't have any sand in it, but we're calling him a Sandbox, but it's the it's the Dome. It's a flutter Firebase tone by providing feedback
on the website here. And if you didn't like this presentation, my name was Coleman canvas, and I was from the Chromecast team. We finally get that coffee. So we totally do going to head off here and we had a little thing popped up in the end and we told her there's no doubt. You're going to meet us at the sandbox. That's right. Go away. Thank you.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.