Duration 36:03
16+
Play
Video

Drawn out: how Android renders

Romain Guy
Senior Staff Software Engineer at Google
+ 1 speaker
  • Video
  • Table of contents
  • Video
2018 Google I/O
May 8, 2018, Mountain View, USA
2018 Google I/O
Video
Drawn out: how Android renders
Available
In cart
Free
Free
Free
Free
Free
Free
Add to favorites
21.63 K
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speakers

Romain Guy
Senior Staff Software Engineer at Google
Chet Haase
Leads the Android Toolkit team at Google

Romain is an engineer at Google. He worked on the Android Framework team, leading the UI toolkit and UI renderer, and now works on new Kotlin and graphics related projects.

View the profile

Chet leads the Android Toolkit team at Google, which works on UI, APIs, graphics, text, and animations to help Android developers create rich applications.

View the profile

About the talk

Come learn how Android turns your app's UI into pixels on the screen. Understanding how things work under the hood can help you figure out how to get the best performance for your applications.

Share

Hi Welcome to our talk drawn-out how Android renders the UI. It was almost called something else. We called it this and then somebody in some Administration position decided. It was actually going to be called how to optimize your app for top rendering performance or something. That wasn't what the talk was about to change back inside since we're going to tell you the details of how actually stuff works. I'm chat Hassan from the Android toolkit team and then will my team and I do real-time graphics. And that's kind of what we're

talking about today. So we have given Bruges ins of this talk before and we thought we were done and then we realized that enough stuff has changed inside the system that maybe it was time to go through this again and see where we're at. Stop. This is our attempt to do that. Let's go. So first of all, there's this word wondering what do we mean by that? Normally it means melting fat in order to clarify it. That's not quite what we're going to talk about today instead. We're talking about the process of actually turning all that stuff like buttons and check boxes and

everything on the screen into pixels that the user can look at and there's a lot of things going on. There's a lot of details that we are glossing over today because we only have 40 minutes to do this, but we'll dump a lot of details on you along the way. So first of all, I'm going to take you through a few for a lot of the rest of the talk. I'm so I'm going to walk through sort of the life of what happens in the flow of information down to the pixels on the screen really quickly. We have this thing called the choreographer and not kicked in usually 60 times a second and it says hey do

you think this is the interval in which the frame is being synced? Buffers flip onto the screen it's a good time for us to process a lot of information and then handle rendering that information as a result of that. So we get a bee sting operation that stand up to the Java SDK lands and Rhonda UI thread and all the sudden we need to process input event, which can trigger changes in properties. We also run any animation. So we change property values again that may trigger things like we do then the whole traversal pass which is measuring views to figure out how large

they are laying them out, which is actually positioning them to where they need to be and then drawing them once all of that information is done. We think the result of that information over to this thing called the render thread And renters red pics that and says, okay. Well, I'm going to then execute these I'm going to basically turn these into native versions of all that information. We produce dobbelare and then I'm going to get a buffer from the gpus that I have a place to write this information and then I'm going to actually issue. All these GPU commands is opengl stuff over there.

And then I'm going to say OK it's time to swap the buffer and then I turn it over to the GPU and then the graphic systems does something called compositing and we're going to talk about most of these steps today picture. The little colored dots on the top of the screen will be sorted noting where we are in the process as we work through a few examples so that we can understand this better speaking of examples. Here's a simple one. So let's suppose that we have a user and the user clicks on an item. I wrote this awesome recyclerview application.

That looks exactly like this. I know it is Because that's a screenshot for my awesome application. It's a recyclerview with a bunch of items in it. And when the user clicks on when did the amazing thing happens it turns into a random color on back. It's incredible. I could give you the source, but I don't know. It's pretty complicated. I'm not sure you'd understand it quite a bit and then I populated it runtime with a bunch of random items in there basically like this exactly like this.

So you walk down from the decor of you and you got it later layout in frame layout and I'm exactly you sure why that we have the deed nothing there. But whatever history we have a bunch of stuff for action bar in there. None of that really matters. What we are concerned about here is what's actually going on in the content hierarchy because that's what you can affect with your application to have content framing layout. We have a constraint layout on the outside wrapping the recyclerview and then all of the items Play these are the items that are on screen because those are the

only ones that are actually being measured and laid out and drawn. All right. So what happens worlds walkthrough this example and walk through that entire flow that we went through at the beginning so user clicks. There's a bee sting operation that gets done up and we process input during the input phase and we noticed that this is a click I'm glossing over some details here actually would we're going to notice first if there was a down and then there was enough and then it gets prices The Clique just take it for granted eventually going to process a click here that I haven't met

amazingly complex example and in there we're going to set the background color on this item to a random color. That's why I call the method random set background color in view. Java, which does a bunch of stuff to set the color on the background drop off and then eventually called this method called invalidate invalidation is the process. It doesn't actually redraw the views. It's the process of telling the view hierarchy that something needs to be redrawn. So you got to click that happens on the item down to the bottom. So that item to you

see it surrounded by Green. We have a little invalidate method that gets called on that and that basically walks up the tree. It's called a series of methods all the way up to trade because the The View needs knows that it needs to be redrawn but it actually needs to propagate that information all the way up the hierarchy so did then we can redraw all the things on the way down later. So we call invalid a child all the way up the hierarchy that eventually ends up in a massive class that we have called do root in full. Java and we have this invalidate child method there and that basically

does nothing except says okay. I need to schedule a conversa. Which means okay. I'm taking information that somebody got invalidating somewhere. That means that I need to run my traversal code later at the end of this process. Traversal is the process of doing all the faces that are necessary for actually rendering that frame specifically, it does measure how big the views are layout setting the view position and size and drawing of use. All of that is called traversal we scheduled time and at later time is now so in the same frame we end up in the

traversal code and has performed reversals method. It's going to do a perform draw which ends up calling a draw method on dachora view and basically that's going to propagate all the way down. So the drama method actually end up in an optimization that we implemented back in money. Come called get this playlist is a structure that stores the rendering information, right? So if you look at the way the button code is written or you know, if you code in general we call Graphics commands and canvas. Like I don't know draw backgrounds draw draw will draw line, whatever. But these end up as

operations in a display was this is a compact way of representing those operations as well as the parameters for the offer up to operations. So we called get this playlist added to court you did not change in fact, so it's going to say well I didn't change but I can certainly get to display it was for my child and then on all the way down the tree until it gets to item to and it says, oh I did change like when invalidate was called on me that triggered something so that I know I need to redraw myself. So get the playlist actually ends up being a draw call on The View where it get rid

regenerate its own juice playlist. So now it's so ends up in the salon method that ends up in the operations in his flight was for that just played with consists of you know, for this item basically wrecked information and text information pretty basic and then we have the Des playlist for basically the entire hierarchy, right? So it wasn't just the view itself, but we have the view hierarchy itself is reproduced in this hierarchy of This playlist all the way down. So now we have the display was for the entire tree and that's all that we need to do on the UI thread. Now, we need to

think that information over to the renter thread in the render thread. Is it separates Reddit actually dealing with the GPU side of this operation on the driver side. We produced all the information on the negative side. Then we actually produce we take that information then think it over to the GPU. So we have to think operation. We're basically we copy of handle over there. And we also copy some related information the copy the damaged area because it's important to know that that item to that's the only thing that change during that frame which means we don't need to redraw anything else

outside of that area. So we're going to copy over the clip down there so that we know what needs to be redrawn and we're also going to do some optimization stop like uploading bitmaps. So this is a good time to do it at the beginning of the frame give him some time to actually turn those into textures along the way why would any other stuff? Is a new type of business integration that was added in an ortho. So typically when you have a big map where to allocate the memory on the jealous side, and then when it comes time to draw we have to make a copy of the business on the

GPU. Is it expensive to text? I'm a double the amount of ram were using that are available in or you you can keep the genocide of the equation and you can have a beat that the teachers only comedy fights did not ever again. This is a really efficient way memory-wise to store your big Maps memory. We have mentioned that renders read before this is something that we introduced in the Lollipop release. It is a separate thread that only talks to the GPU is native go. There's no call out that the job because there's certainly no call outs out the application code. It just talks to

the GPU we did this. So we still need to do basically the same thing we did pre-render thread, which is we produce other display with information. And then we think or we we send that just played with information to the GPU so it's sort of cereal but the renter thread is Able to do things atomically like the circular reveal animations as well as the Ripple animations as well as Vector drawable animations can happen atomically on the rims with red. So that's something that can happen without installing the UI thread. And in the meantime, the UI thread can be doing other things

while it's idle after its think like maybe some of the the idle prefetch work for recyclerview with it was done last red just red ticks in weave sync everything. We have this playlist. We have the damaged area and then we do we turn the display was into something that we called DL Ops. I'm just play with operations. So you can see that we have that fill operation in the middle of that something that we turned green there. And then we have some optimizations that we perform. So we we we do here is her phone since you should do alpha rendering Bitcoins to tell him if you or she said

to Hardware later on The View, we try to identify to join commands that I need to be talking to those layers and we moved at the beginning of the frame these supposed to stay changes inside the GPU which are extremely expensive horrible horrible performance waiting for the instruction example investing we look at all those operations and you can see this example because we insure leave a lot of operations at the stupid it again here will changing the state of the Jeep you several times. But instead what we can do

if we can draw all the rectangles together all the text together as part of the real during investing. That's what we do. We say look if we see a bunch of Texas was the same color in the same font. They don't have to be different draw text call to can you just single one that covers the entire screen? You can see who are the original dlf pads a fill operation and then it wanted to draw some texts, which is going to be end up being text format copies from the glyph cash. Then it's got a fill operation and then some more text in a fill-in. So we've got all these operations in early. So

after the reordering operation, then it looks a little more like this. We got a series of bills and a series of texts operations. They can even be back together to be more optimal which we will see here. So this is an example of Gmail's that was in the honeycomb the area you can see how she would you find a pipeline to slow down the rendering and you can see exactly how Gmail was drawing. So we have a lot of this item and we just call them in the exact order that they are the

best thing and merging and real dream we get this you can see in particular that sold the stars were on at the same time and most of the texts appear all at once. What's interesting is we we are drawing all the states and backgrounds one after the other. So that's good. The real during worked the best things in the work we have to draw them one after the other 200. Make sure the alpha that used a correct. Depends on the application in Kit Kats to settings application

was we could draw the whole screen in about 6 drawer calls instead of dozens and dozens of your heart condition for us. I think this work at the time on current devices save something like a millisecond which doesn't sound like much unless you realize that we have to do everything within 16. There's actually a huge Improvement that allowed females to be less cranky cuz then it wasn't pushed out into the next Ram is often a jack. So this is where we feed in the information about the damaged area. Right so we know where item to was on the

screen and we know that we don't need to draw anything outside of that. So as we're processing these DL Ops, we know that we can basically throw away anything that is drawing outside of that area and graphic that's called a trivial reject the way trivially reject all the Dale off that weren't intersecting with that area. And now all we have to do is draw a fish. Oh and some text in a line. So we do that in the process of doing that we could do a get buffer. This is usually an implicit operation. We don't request a buffer. It's more that as soon as we start doing GPU operations.

Then the GPU hands us the but more particularly surface planer enhances the buffer that we can then put these commands into then we issue the command. This is a series of GL commands as you can see on the slide. It says DL command, basically the equivalent of what we need for whatever doing the pill with the text bitmap copies lines, whatever and then we swap the buffers to this is us saying we are done with all of her rendering operations. We're ready to display this frame on the screen as a request a surface planer to then swap the buffer. Basically we're done trying to the buffer you can

swap this with the one that's in front and it'll put it on the screen. Meanwhile in surface planer then. We have a composite stuff which reminds going to talk a lot about later. But basically it takes all of the windows on the screen. We see here the navigation bar the status bar and the actual content window for our application. It combines all of those in the hardware compositor puts them on screen and then we're done. All right. So that was a really simple example. Let's look at a super complicated example. This one's going to be in two phases one. So we're going to drag the

list up. So we're going to drag it. And then as we dragged that we're going to move the items a little bit and then eventually if we keep moving them we're going to have a new item up here. So we're going to look at two versions of this. There's the move only version. So as we dragged it up a new item not a new item appears everything just shifts a little bit up. So first of all, we need to process the down. So we have a feast think it says time to process input event. So we do that and we end up in code like this on touch event in recyclerview. It says there was a down operation and all

it needs to do is register where that down happened. It doesn't need to process anyting nothing changed on the screen. We just register that the user actually press down to record that for later and there's no off. We don't do any of the rest of the stuff we talked about cuz nothing changed. Alright, they keep dragging then we end up in similar code. So we process input on the next frame we say okay on touch event. Oh now we know that they've actually moved when we know how much they move because we save the old X and Y and we calculate the Delta and now we call this thing called

offset top and bottom. We basically for all the views on the screen. We simply move them in y and offset the bottom called something. It's an invalidation method, but it's slightly different. It says invalid 8 U property. This is an optimization that we put in in probably honeycomb II release or something with this playlist properties. So when I talked about this playlist earlier, there was one Nuance that I left that we have the information about the operations in the parameters for the graphics operations, but we also have the information about gorgeous playlist properties,

which are basically properties of you like the translation property rotation Alpha and these are properties which we don't need to re-register the view to change we can see We change them in the display with structure itself and then they get picked up at GPU issue time. So it's a very fast operation for us to do that. So instead of invalidating of you and redrawing everything in that view. All we do is say change the translation property in this view. So the way that happens is we call invalidate View Property that propagates all the way up the tree cuz we still need to know what the

top layer what happened but it's a much more often. So this ends up in schedule traverso's as it did before in the draw and ends up and perform traversals, but perform draw can do a much simpler version of this because the display with didn't actually change all we did was change this playlist properties inside of it. So we can immediately think that information over to the renter threads week and then execute that turn that into this playlist tops get the buffer basically everything is as before second phase of that super complicated example user keeps dragging and

as they dragged a new item appears on the bottom. So do you think we'll try set the input we end up in a method something like this? Okay, we know that they moved. Oh, but that means that we need to trigger The Creation in the bind of that new item there that ends up in code like this where we actually add of you to the parent. So the recyclerview is going to get a new view and it's going to call request layout soap request. But instead of saying I need to be redrawn it says I need to be re-measured and relayed out and that could side effect everybody there. So we

basically propagate a request light on all the way up the tree like invalidation and then we do measure and layout on the entire tray just to see what change their so request layout happens when the parents and then that propagates all the way up. And that ends up again and scheduled reversals our friends and then perform traversals. We're not talking about to draw stuff. Now we're going to do it for for measure and a performing layout measure is basically asking all of the views how big they would like to be. It's a request and then layout says this is how big you're going to be in this is

where you're going to be physician. It's a negotiation between the views and all of their parents according to all the constraints in the system. So we do it for for measure that basically calls measure the top and that propagates all the way down and then we have all the information about how big all of yous wants to be and that is good enough for us to calculate the layout information than we probably get laid out all the way down the tree and once that happens on the item and the parents are changed then we actually lay out that item and we're ready to go now we can actually draw things.

And everything is as before. So the new ones here was just delay outside except unimportant nuances. We're talking about all of this request lay out and measure and layout happening in for this recyclerview situation. However, recyclerview optimizes this it knows enough about its own parent and its children that actually can just offset the views instead of doing the request. You can actually just moved of use out of the way and create the new item optimization for recyclerview as well as the older list you as soon as it's all the windows

on-screen. This is interesting or Festival because it's always interesting to learn something new about technology understand some Concepts that are behind some of the public apis leg so fast the first exterior surface view have to understand Refer to as the name suggests. It's just a q of buffers there. Where are crashing stuffers. Lou going to have one two, three buffers the other option since you only work when we set up a quest how many buffers we want has two endpoints. We have the producer and we have the

consumer. So typically the way we use a buffer Q the producer called a message call Dick shoe buffer on the cube grabs a buffer from the Cuban owns it you can do any kind of rendering can be sitting the pixel data directly can be using it doesn't really matter in one use opengl. This is basically what happens when you pull suede loafers at the end. That's when we'll put it in the contents inside the shoe buffer and gives the best for back to The Chew consumer using acquire. Sinkholes

choir buffer you fix this first available, but forensics you he does whatever it has to do with it and when it's done it puts it back by pulling release. Such a pretty simple concept of roaches rats look at the code in the overhead of files and all the nations involved in part because the two endpoints of a perfect you can live in different processes and this is exactly what happens. This is how I was the first computer works. So when you cry in the system, you have the window manager and yourself, a swinger flinger is all consumer

guide you and this is what's done automatically for you when you create a dialogue when you create it to stay believe when you create an Xfinity that window object has a sibling on the surface king of side called earlier with buffers and shoes and that's all we do and we've run out of names and their Graphics team thought of you in later and window and Window in the lair is the confidence in the system that creates an owned the preferred fuel for your applications with Christopher q and we have a

way to send the endpoints to your reputation the cringiest office. So whenever you see the surface in one of our apis, you're really have the producer endpoint of a perfect Cube that lives somewhere else in the system either new process or in some other process most of the time it's going to be in search of a swinger. So now when people use case for you as a significant of the application developer or you're going to be here, then we cook a horse with a surface

effectively and we asked the manager and surfaceflinger to create a second surface and we just flight attendant and we pretend that the fact of the same window will not have two different different surfaces that two different perfect used in a completely independent from one another jail or volcano media player. Use a surface texture. So you'll put you sure you saw your consumer will be open GL. So you create the stuff that sticks to your back. You need to see

a picture ID create sand owns the different you so that you will often be in your own process. Then you have to pass Yourself by creating a surface is a constructive surface surface texture. You create yourself as you send it to some other application and then when you're close require to get to suffer from the from the effort you it doesn't turn to ring inside another Cube and then when it's done, it's been 4 release. Fix'd review is the widget the Spotify app that you can use to benefit from surface. Surface texture is the

consumer and you still feel responsible for getting the stress test from the texture of you and giving it to a producer of your choosing to be using acceleration to tell you that picture of you. Why is the solution into the surface you when you wanted to integrate a video or opengl rendering inside the complex application Swenson street view or child you or anything that was animated Windows was not your decision for that. It was not synchronized rendering of education has been fixed in the recent versions of Android. So most of the time on recent

versions of Android, you should use this refers you instead of a texture you use a text to view only when you have to maybe set Search between the other views or using animation that's not supported by the surface. And what is the different for this reason consumers in the platform? Surface texture opengl? Yes. He's a producer or a consumer that I suppose we get a buffer. That's when we told each you buffer from the render Smith. And this is done. Typically when we do the first draw call and frame. It will

actually told you the frame and put it back in the buffer tube. You can also use things that Vulcan to Media Player indicted. I can you have many many more throughout the fat for me. Not the actual composition do we have created multiple windows to each have their own Lair surface finger knows about all those layers and to talk to the displaced temperature is a hardware abstraction layer that we use because we want to avoid using the GPU and we need to complete all this Windows on-screen one of the reasons to save battery smoke

Shisha that's with grease also to make sure that your application has access to basically all the capabilities of the GT. We don't take it away from you. And is it bad to have heard that the screen and you'll see why in a few slight so we have to wear that that's really fast at the king multiple beat them together on screen. It would just talk about this actually works to have a composer is really a can of a particle. I'm going to describe the old is

wonderful. Composer to it's much more complicated. So I'm not going to describe is here. But the gist of it is that you can work the same way and composure and we're going to send a older layers to go have a composer and asked you to tell us what he wants to do with every layer every other temperature using and there is no way that we can ride the drivers for all the different color composers are there. So instead he wants to do the things to have a composer understands the pixel format of that window and tells us that you can handle it and where do the

composition for that window so we should going Steines overlay for the second layer telling us over. So that's great addition can be done automatically Forest on our behalf. The very cheap way to now send older two years to the Huddle composer this time for a position and do everything to the screen. No more complex example. So we have the number of players we go prepare for the first one. Everything goes fine. Composers. His overlay can handle it for the next one test. So that didn't happen. When you using Excel

format the first reported it could be because you're using a rotation in the other doesn't know how to handle rotation or we have too many layers on screen or any number of reasons the composer of this was much more common in devices probably three or four or a long ride route to go to have about four layers that you could use that's for the type. That's what we're used to at 4 and we got going into too much detail you have as you can see 7 used to be cool to excel we used to have those layers to draw the rounded Corners so you don't forget

F7 EFI. I need to know about the details when we have layers that donut Handle by the Halo composer. We need to use the GPU who composed them ourselves. So she has to be able to do everything that I can do any situation. We need to create basically a clutch buffer another layer in a format that we know the Halo composer can accept and then we use custom opengl code to do the composition of tales of the two layers. So then we're left with only two layers and we know that just going to send

it to the other composer. So that's what we do we call set. And then you show up on screen if you if you're curious sometime. You can run this command ADB shell dumpsys surfaceflinger and socialist capitalist great importance and it'll spit out way way way more information than you want. But one of the things that it's going to show you it's a table of the windows on the screen and whether they are currently represented as overlays or frame buffer what can happen sometimes is

if layers of Enos mean for a while and We Know download changing the eclipse in them into a single layer until the change again. So the output of the command is sometimes a little bit misleading cuz you might be seeing the result of accusations based on time. So the best thing to do is use which one this when you're running an animation of something is changing on-screen that's going to be the most valuable information for you. Tell you to use the giant of invalid data text Architects are rectangles only part of you that you knew need to be repainted That was supposed to

play with older Android devices because been with was extremely limited while using software rendering and even in the early days of GPU rendering for us, we were pretty easy. If you do need to do this anymore and you turn on recent version of Android before this was different. It was extra ignored by the system. Now what happens every time you go in there or they needed for the right thing on The View reminder that hold you or equal to Deatsville damaged area. You don't have to worry about it in one of the reasons of getting rid of it is not

necessary anymore for savings pretty easy to have enough by one error or rounding error in to get a C-section screen and so many bugs Around that. So now you don't have to worry about it is now able to do preaching of ice and a head of time that way to mention this earlier. This is one of the Winds they were now getting out of having a separate render thread because now well there's a title time DUI thread was done with his work after it sank looking used at idle time

productively for doing other things like fetching things that it know it might need in the next few frames. You can use phones. That's what we use when we take a screenshot or when we record a video or when we do casting Chrome casting for instance of his finger to perform a composition with not directly just into another surface, which is another way of producing a surface Excellence simplification grafica with a K available on keto it was written by a member of the grass extinguished years ago collection of oldest things you can

do View with the media encoder. We just fill this place. It's very interesting piece of ktulu cats, which will discover Management in that's one of the things like night-light color transforms. We also have color blindness simulation tools can be handled by the composer in specific situations and the Zoo cost for full performance issues iPhone since a while back Night Lights was not reported that Billy Von D n v x or than sixty one of the reasons was the

hardware we need the drivers have to be to call a transform. So we had to go back to default position was really expensive. It was hurting the battery so we can have the other two dice and I was just at a high level overview we give the number of stalks ended by what we do functions in the UI Runner rightsell how we do the best thing in merging the station switch your interest Shadow calculation is interesting. I think we had a talk where we talked about some of those details, but exactly where does that fit the rendering pipeline

is hurted a clever, but yes, lots more going on there, but hopefully this gives you a general sense of how things work on Android and thank you.

Cackle comments for the website

Buy this talk

Access to the talk “Drawn out: how Android renders”
Available
In cart
Free
Free
Free
Free
Free
Free

Access to all the recordings of the event

Get access to all videos “2018 Google I/O”
Available
In cart
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “Software development”?

You might be interested in videos from this event

September 28, 2018
Moscow
16
159
app store, apps, development, google play, mobile, soft

Similar talks

Marc Bachinger
Developer Advocate at Google
+ 1 speaker
Andrew Lewis
Software Engineer at Google
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Patrick Brady
Software Engineer at Google
+ 3 speakers
Dean Harding
Software Engineer at Google
+ 3 speakers
Rasekh Rifaat
Software Engineer at Google
+ 3 speakers
Lauren Wunderlich
UX Design Lead at Google
+ 3 speakers
Available
In cart
Free
Free
Free
Free
Free
Free
Fred Chung
Developer Platform Manager at Google
+ 2 speakers
Dan Galpin
Team Lead at Google
+ 2 speakers
Eric Kuxhausen
Software Engineer at Google
+ 2 speakers
Available
In cart
Free
Free
Free
Free
Free
Free

Buy this video

Video

Access to the talk “Drawn out: how Android renders”
Available
In cart
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
558 conferences
22059 speakers
8245 hours of content