About the talk
There are many options and technical details to consider when building a performant and scalable data backend in the cloud. This session will help you understand the various storage options available on Google Cloud Platform, along with tips and tricks to help you make the best decision for your data backend.
All right. All right. Tell everyone. How are you doing? Good. I hope there's still some more people getting out from the keynote. I really appreciate that. You came here to watch my talk. I going to talk about the powerful data back end for mobile and web little bit about myself. I'm so basically I'm the interface between the people that built the technology inside of Google and people that built on the technology outside of Google. So hopefully you guys and girls going to take something out of this talk and feel something amazing afterwards.
Before I start I want to get a little bit of appeal. Who am I talking to so if you hear is it's a developer. All right, that's amazing. You're in the right place who is an Android developer with Firebase? All right. I'll have some awesome. Let's get started. So if we choose at the Right Storage, it is actually really really big problem is it's really hard to do. We have a lot of things to consider when you choose your story the things like what's the object size? What's the scale that you need? What are the latency requirements
throughput consistency and like permission sings like that so that they can you to consider when choosing the right Source system. Now when I created this talk, I was thinking like what am I going to talk about it or how do I going to construct this talk? So I was wondering if I should start with something. I have to get it out 20 years ago pretty much storage systems which are mostly local. How do you say even started with the Fife and Aquatic assault and biscuits but they're pretty amazing
thing. It's really interesting and everybody should look into that but we're living in the now and almost in the future so I can talk a little bit about the things that you can do today. Okay? I'm going to do that. Like I didn't want to go with sore like a try talk in a sense of like talking you through the different options that you have like the Firebase Orvis Google Cloud platform. I want to make that a little bit more entertaining. So I thought about let's let's do this and trade in Temple Ave. Okay,
start up pictures concentrate on one thing and do one thing well by the end of the talk, I want to have an app for top load pictures where you can crop and resize the picture where you can stop me now and then of course, please play today at the pictures. So what do we have in terms of options recommend to look into that fireplace keeps you a lot of things a lot of tools to build applications, especially applications. Like Android iOS or replication indication for staying fire store, which gives you a flexible data storage for you have to
attend executor was I could call for fire base? That might be situations where where you have to get out of this where you want to do a little bit more sophisticated toss things that you can cover all imitations of malfunction and there are reasons to to break out of this ecosystem. So they can use if you have compute from like brvm, so you can do pretty much everything that you want almost up to Cloud functions as well. Night, so you have a multitude of computers that you can use to
do your things. We also have a multitude of storage product under the Google Cloud platform portfolio same product that you can use with Firebase. So you can think of that you have different interfaces to the same product which makes it actually easier to deal with them for sisters different use cases. So if you build a web app, it's easier to use to Firebase SDK if you build to a banking system, you can use layout line libraries for on the Google Cloud platform. So the first thing when we upload a picture, we need a place to store our pictures. So that's why
I said they're both the same. The product is actually a higher the available throwable simple and Secure Storage. What does it mean? What is an object Source pretty much every kind of block that you can take off a plop is basically bites candy images of candy videos files. I can pretty much be any fight stream that you can find. Note to self storage is optimized for for you to latency for Andrews latency there a lot of things that you can do. You can Define the caching
strategies things like that. You can make you object Storage Public so that people outside Connexus you can actually make it in the sense that other people pay for the excess to your cloud storage system. Get into creative / 144 Cloud Park City Hall. So basically if somebody comes in from let's Asia or Europe they tow it to the closest pop five richest connected to the internet service provider and then they go directly into our Google Network and that means latency to retrieve objects or you
can store up text minimize. So it's really good to go to say that you have to create to be able to store some data when you create a bucket you to find things like the storage class and the location where your package lips. I have two examples. I thought you use our cliches you told you make it back at you define. The storage place in this case is small TV channel, and then my storage is cloudy to pick a cloudy day pictures. You can do the same with Java if you want to so if you want to
create packets for the medically you can use our storage and then basically directly created from your Java program. Next step is uploading a file again, very easy, if your family with CP on any kind of operating system, you just pretended with cheese you withhold HCM, which allows you to like a multicast. You're here to our clocks. Pretty much simple and straightforward as well. You select the file. You point to a file you make bikes 3 Molotov by Terry out of it. You put it in
the bucket and create created with the name. As you can see here in this example actually also gave it the content type that you can do as well. If you want to do it from that for the ones that have done fireplace, this looks familiar. So basically you use to Firebase SDK you get the reference use you as a file reader you get your from your input field you get the file name in the content and happy with that to to storage very straightforward. The next last but not least for storage is how
to read a file. And again, it's very straightforward. You can do a couple you like download it. You can use Java object and read it from from your pocket won the rap you basically I got it download you were all and then you use this download download URL and put it in your image tag or your background image CSS. All right, so that's great. We uploaded all our pictures or we uploaded one picture now, how can I get through processing? Why do I upload a picture from my phone or from my camera? These pictures are large. They're like megabytes
and I don't want to serve megabytes of pictures always to to my web page for to my users that were trained the band with the what I want to do is I want to actually trade some some nails now, there's a possibility that you create something is already on your clients, but I don't think that is a good idea. There are many reasons for not doing this. Now I can create some Nails on the backend which I going to use and I could trigger the something equation from my clients. I don't think that is a good idea either what happens if you upload a picture and then you get disconnected or the
equation fails and you need to retry it. All this logic would be cluttering your your front end and your your client. Would you want to do the thing on the back end, so I want to use a different mechanism to actually triggering the processing of my image files notifications come in who is cloud storage allows you to basically register a hook which gets called if anything happens on your storage system for vent pipes and you can hook your systems up to a title finalized which is basically the object create.
It's an object. Delete. It is archiving of an object and its any kind of metadata update that you do on your Optics. So you can hook into any of these events and their different mechanisms to hook into these events. There are three main mechanisms that can use if you use Firebase Cloud functions, you can use the cloud storage trigger for class actions. He if you submit your function, you can say this functions get hurt if anything happens on my pocket. This is actually and I need a creative if the cloud pops up notification system. So
if anything happens of these events that I declare, please send me a pops up message and then you can't connect the system that deals with the pops up message to change notification that are still around. This is what we need to do is very early on in the cloud storage platform storage product is the optic change notification. So if you were just enough to change notification you basically get an HTTP call turn H-E-Bs endpoint and then you can deal with the message. They are small thing about this. Is it in the way that it
does all the retry. So if you don't acknowledge a successful processing of the message as assistant just reprises automatically after 30 days. Type. So how do I set up a cloud pubsub trigger an example are using the speak loud pops up SDK notice two-player a notification light right here in this case and notification against my cloudy pictures pocket and I say I only want to have notifications when object finalize this happening. So I'm going to knock tickets graded and
then I declare the puppet where this notification is going to be sent two messages. I create a so-called subscription which basically hooks to the topic that I just traded. So let me show you how that looks like really quick. Sonic's, are you here? You see my application right now. It's pretty empty if I refresh this nothing is happening. So she wrote this talk. I actually wanted filled it with pictures so I can show you if I upload a picture here. I just select one picture. I got to do a blowout. and it's going to happen. But pretty much nothing else
going to happen. Right? So if I going to refresh my page, this picture is actually going to be gone. So I need to implement all the things behind the scenes to process this picture. The first thing that we going to do is we going to show you how to create this. Let me make that a little bigger. Can you see this? Thumbs up thumbs down that you can see that are cool. So first thing is G Cloud pops up and I know that's right. So I created my notice topic. All right, perfect that happens. The next thing that I going to do is I want to actually create my
My butt command. So now if you have spotted I actually name the topic differently. So I going to remove this year and create this but actually I'm creating another notification. What on my my storage pocket is he here? The storage is named a little different because that is the storage which I actually use for my application. Now the next the next thing that I want to do is I would actually want to see if any messages are coming in so I can do notification. That pops up.
Sorry, she's not. UPS hub notification Hi, so it's up to attend. I just waited this up Cochin. Like I actually need you to create a subscription. I'm done that. Greatest scription. So now I have a subscription that I can listen Legends on that topic. This is a little slower than let me turn off the Wi-Fi and hope that it works will be better without the Wi-Fi. So now I want to actually listen to my subscription. And then I can pull. And I going to pull I'm not going to acknowledge, but I going to pull an uploads.
So I can see you right now. That's not nothing going to happen in CR items. So let me upload a picture really quick again. an outlet that picture and then going to go back to my system to pull up. as you can see I have them have a message as you can see there's a lot of things there's things like the idea of self-loathing for its name the Buckhead Generation IV, all these things that you can use Hi, so next thing let's go back to this lights. if my my pocket so now if I upload my pictures and protesting them, I won't actually store something around this. So
even if I now process my pictures, I know. Nothing's going to happen. Basically, they're not be displayed in my Backpage. Do I need something to actually process them and then stores them made. They are anthem of systems that you have available for us and she is going to look at Firestone tire store is an OS system which document based data being sent. So if you want to admit to some some peel so thankful that you can do without that throughout the life cycle of the application. So you
don't have to be very restrictive on the steamer that you were going to use. You can just Waltz the scheme of while you were at the application. Norfolk fire store getting bigger and Library Street, you can use from the web as well as from go back and systems. So that's why our Stars just to summarize again. You have an expressive gray language thing for a language you get real-time updates. If you fire base you can basically use watch mines. Are they to get events for any theater that has changed application and it's designed to scale again. So this is a system that can
kill to message sizes. How do you write state are to Firestorm here's an example for job on pretty much everything in fire store is acing so she can see here. You get the document ref you give it a name for your documents and ID document and then you put your fields in there to write. This is an API future and then you can also update Fields if you want to know is if you replay. It's pretty much similar to that. They're not a lot of changes here. You can just do on the document draft that you create you just do
it yet and that locks till you have today. Type. So let's see how we can integrate that today actually write some theater in our application. So now he's going to switch to hour to some coat. So what year is my image processing service here is I basically set up my application with Google credentials. So this uses the default credentials that you have set either if you're Undead on a TC instance, I compute engine and since then you have to fall con text or you provide a service account, which is used to authenticate. You need to
get the storage container to you get the Prudential contacts check ID redefine our storage that we want to use to find hear my pops up that I have to find and I can actually use this and say, okay. I created it pops up which is uploads so I can I use the Diablo subscriptions that I created earlier. Listening for pops up messages again doing this isn't going to sleep every time I get a message. I have your my message that I get I going to put that into my
process message function which we going to look into in a bit and then if the functions actually successful, I acknowledge this message pops up is if you can actually have multiple subscribers on the same subscription and what happens if your message get round-robin distributed to all the types of drivers on one subscription now if you want to make sure that one message is this route to multiple places and always to multiple places then you just create multiple subscriptions. All right. Let's look into our forces message. The one we going to
do here is basically I going to check for uploads on my on my image when it gets uploaded and I can show you really quick you happy hour storage system. If I go to storage you going to see here. I have some images that are the two images that I uploaded earlier. So let me type and then I basically take the pots take the user ID out of this take to file ITR out of it through some resizing. I'm doing the resizing geography imagemagick. So that's the reason I wanted to run this meeting since Winston eye stores the sums in my pocket that you can see here in the
create function. I have like a 600 pixel wide some Neil in the 1600 pixels wide thumb nail. Just great and move this object as well to a different folder that it's not in my uploads folder anymore and clean up all my my temporary files that had a head in between. So, let's see what's going to happen since we uploaded some pictures earlier. On this on this weekend sees as you can see now it actually gets these messages and does all the processing. So let's have a look how that looks like. So we see here that we have our
messages you can say on my storage and see if anything's happening at 8 and raw images now, so we have the images moving crate amazing and let's see if any man is coming in. Natalia is not coming. So let's just upload another picture and see if Alameda theaters coming in. I'm not writing any way that they don't actually needs to write my first so we need to look into setting this app for writing Samantha data. So the first thing is I want to have my fire store. So fire store in it. So this creates takes the credentials again and initializes my fireplace
and my files for this before I move my data to the wrong folder. Actually. Let's do it died and left after that. I want to update my my fire store. Made it there. So he's just going to add three Fields the process field to Sun by beautiful. And a large available team save this on a restored. my application something is missing. All right, fibrous is modere firestore. Of course, we need actually to provide this year so firestorm. And then here I want to Light My Fire store. Play this again.
So now it is working. So let's have a look if we upload the picture. joblo, the picture So this is actually just a client site that I display this picture. So if I refresh I going to see nothing that's okay. I going to see nothing because it's basically downloading the file doing damage magic and then uploading the file again. Let me see you and you coming in. Let's see if you get all the stuff. All right. Perfect. So if you have knowledge to me today. Their peers means our picture was fast and For a second. I got a awesome. So if you have the first
picture there, let's switch back to the east light. Hi, we have our first mineral minimum viable product. Right so we can upload picture and I can display pictures amazing a cross-bite light lot of people's houses of people are alike starting to upload cloudy pictures I said before this is a very targeted app. Only for cloudy pictures do is I actually one of the two some cost. In the beginning you saw I created the Buckhead which is the multi-regional bucket
like most sophisticated sorts class if you have glasses that I can use. There for storage classes available under the couch storage on is the one that is distributed across two regions. Retiring at least a hundred miles apart from each other and kiss you at the highest availability for any data processing and you want to actually process your theater close to where does living then we recommend to use original Fat Cats in one region is lobo cheaper and gives you a little bit less off and I'm still archiving data we have in your line and a cold Line storage last
if your images or any kind of objects or older, they are not used within 30 days. You can move them to me a line when it's cheaper actually to use and your line than have them storage on every time I'm at original a pocket like less than once a year then it's cold. I'm still have the same. Like persistency of the off your off your data object, and you still have to see me if he is, which is amazing, but the prices get you down all these things automated application
is something called up Tick life cycle. You can Define rules when object it automatically transition between the sword classes. If an object is older than 30 days, please move it to your line if it is older than 65 days older than 90 days, please move at 2 to call line. But I also can't do that throw object. I don't have to create a specific sort slots. You can do that, but you don't have to you can actually change the storage loss pill object. How does it look like you can use it as a 50's
YouTube and do it there or you can use a Java. There are two objects. Are we going to do this year? A second application here, which I expect some more metadata the same like setup code pretty much but what I do is I'm reading the object and then I'm using the SDK to get like exited out of it. So now when this is done, what I want to do is actually I want to move this object of a So, let's see if we can do this so we can actually do like here and move the object.
See how we got to do this. All right, I take my object. I just to find in your storage plus filters and then say like a cave in my data is processed. I just going to move it to me line. They going to start this year as well and it's going to listen on my notification as well. So now this is a topic where notifications has to have consent before so I said can see it starts actually processing as well and it might throw some errors because I have to eat at some some uptake. But it
basically evil process in the background our our object. So what are you going to have here is like you have all objects in time what you can see here is made of you guys showing up. I saw this kiss me all the exit 8 on which I now can use to search for and send or build a world map where I can put all my pictures and if I go to storage which I going to do here. I'm going to go to my storage system going to go to a cloudy pictures iconic go to my raw. Going to see here. And if you can see my picture is actually now have to
storage system align. I'm already moving my story at my object into a different storage Las which is really powerful. Nice, so we have to have to be happy with yourself by moving objects and savings and costs are the next thing that I want to do is and Naval a business mobile. I want to I want to generate some DFS Rincon viral. I've sort a lot of pictures and now people have Austin and want to actually like prints and frames of buy a frame and ship the pictures. So I want any of
this like I'm dealing with money system. So I don't want to do this with fire store because I'm not sufficient for what I want to do. So that's where another source system comes in and we have managed store system is called for cloud FL system Reddit can have my scale War post-credits at the flavor and the automatic updates you can you can do fail over so you can do fail overs in 2 hours. People clouds you can do will fail overs into on primary systems things like that. It is vertical scalable. So you can change your notes sites throughout your life cycle of the application and you
have things like a 99.95 LCS Cloud SQL instance rate. You give it a name you say the TR vs. Paisley the size that you want to use any region where they say about SQL instances braided next thing is I want to talk to my cloud SQL. Why does actually and a proxy which automatically outer indicates you provided a service? So, you know what you can see is your face me telling you what your Cloud SQL proxy and then you make an executable and then you can run it either locally or you can run it on your TV instance or if you are coming over which is a container management system. You can run it
directly in your parts as a sidecar to connect to the cloud in instant tan something with my scale or any SQL database amazing. So this looks very similar. relational schema and creating a database and some tables In this case, I graded you was you was a table the cloud Aficionado migrating about unstable, which is facing my account balance and I'm craving a transaction table which shows me the history of the transaction and then I can do things like a like a distributor that
actually takes the money from the balance table independent stable and also writes in my contacts and table that that happened this day forward. Not an expert base. My apical is viral, right? So so when I pulled my cloudy cloudy picture 3.0. I want to be to reschedule a bowl for large amount of people. Log parser or fire store on the fireplace. You don't have to pre probation any resources you basically just use it. If your application and two more you use digital scale. If you needs you
can scale it up to certain point basically the biggest p.m. That your cloud provider or Noah case Cloud platform provides you but if you run out of the size of your of the biggest p.m. You kind of stuck there many many things that you can do or you can look forward to chatting and feels like a distributed dbms, but it's very challenging and cost. A lot of time investment is very risky. That's not where I want to block in Cloud. Spanner Banner is our horizontally and distributed relational database system and its scale special needs you can
scale 2012 notes and scales horizontally your schema permitting if you want to learn a little bit more about lot better if there's a Shameless black at the at the bottom. So if you search for Cil about Cloud spanner, you'll find me. Do you serious right goes through some of the nitty-gritty is off an amazing things that we do have a cloud spanner. So let's recap. What are the portfolio on the storage platform that we have in terms of date on Sports Products scale. We have mentioned we have
mentioned plow Terrace or fire store. There's a couple of us here on the on the slide. So if you could scan use went to that Benjamin fish if you use if you have a lot of data with you want to search really pass the process will want to have like a very high weed through Fort Whedon writes report for non relational data. Then we recommend to use cloud bigtable so you can connect it to your bike spark or things like that. Then we also have a date of a system where you can go and load
relational data in and crunch terabytes of data with secret. So there's this I sent you very much for for your attention. If you have any questions, please feel free to walk up to the microphones. I'm happy to answer some of your questions and are pleased. Also if you want to follow me on Twitter under at house tea rose tea, and I'm happy to answer any kinds of questions to Twitter as well talk and have it go to Google IO.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.