Duration 34:51
16+
Play
Video

Accessibility for AR and VR

Christopher Patnoe
Senior program manager on the Central Accessibility team at Google
+ 1 speaker
  • Video
  • Table of contents
  • Video
2018 Google I/O
May 8, 2018, Mountain View, USA
2018 Google I/O
Video
Accessibility for AR and VR
Available
In cart
Free
Free
Free
Free
Free
Free
Add to favorites
10.72 K
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speakers

Christopher Patnoe
Senior program manager on the Central Accessibility team at Google
Ran Tao
Software Engineer at Google

Christopher Patnoe is senior program manager on the Central Accessibility team at Google. In addition to managing a team of program managers, he leads accessibility work with the Hardware and Daydream organizations. Previously, Christopher worked on Daydream, Google Play Music and on Google's Technical Infrastructure. Prior to Google, he was an engineering program manager at Apple, Sony Ericsson and Disney Mobile. Christopher has a BA in Music from the University of California, Berkeley.

View the profile

Ran is a software engineer on Daydream, currently working on augmented reality (AR) features in ARCore. He actively participates in the Daydream Accessibility efforts and has a keen interest in making augmented and virtual reality (VR) useful to those with disabilities. He graduated from University of California, San Diego with a bachelors degree in computer science, where he studied machine learning and computer graphics. Previously he worked on implementing user interfaces in Google Earth VR, improving Go

View the profile

About the talk

In this session, we'll cover ways you can make your VR and AR more accessible to more users.

Share

Hi there. Thank you for joining us today with a conversation on accessibility and a rvr. Today we going to have a quick overview of AR & VR just implied them back on for those people aren't familiar. Then what's your snack Lucid design principles and guidelines. And then finally, we'll review different disabilities sharing some opportunities on how to make your applications that experiences more inclusive senior program manager here on Google's Central accessibility. I've been many years working at Apple building Hardware software.

Sony Ericsson making phones even Disney mobile making games. I've been a Google for over six years where I worked on Google Play Music and even Daydream RAR VR platform. I spent the past four years and 20% roll for game accessibility and the past year. I've been able to make my side hustle my day job. And hi, my name is Randall and I'm a software engineer on the AR core team as part of Daydream. I joined a dream last year summer after I graduated from UC San Diego Mercedes computer graphics and computer vision. I have been using a wheelchair full-time now for

two years and I'm passionate about promoting accessibility and daydream and the AR & VR industry as a whole So before we dive into making a rvr more accessible, let's have a quick overview as to what is augmented and virtual reality. Here's how we think about these technology. We have VR where the world is hidden from you and the virtual contents around you VR can take you anywhere and we have a r what are you still see the real world and virtual objects are superimposed on top and so AR brings things to you. And let's talk

about VR first. There's so many headsets out there right now and the industry is growing rapidly at Google we've developed cardboard and the Daydream View and we partnered with Lenovo to make the Mirage solo 3 headsets have a key distinction so that two on the left their cardboard and daydream view they have three degrees of freedom tracking which means you can rotate your head in any direction and look everywhere. The experience is kind of like a movie projected on a sphere around you and you're in the center of the sphere so you can look around but you can't really lean in closer to any

objects. And then we have the Mirage solo with the world sense, which allows for six degrees of freedom tracking. That means not only can you look around you can also move forward back up down left, right? And this allows for a much more natural interaction where you can duck when the Snowballs thrown at you. Over the past few years since Google has launched the cardboard. There's been so many experiences have been created. You probably heard of games like job simulator or apps like YouTube VR. My personal favorite is street view where I can go anywhere in the world. They even have 360

photos up on the International Space Station and there's also artistic apps like still fresh and experiences for meditation training and so much more. Endless now move on to a are where we can bring things to you. So Google's arcore SDK, which just a large this year unity and unreal you say our experiences cover similar areas to VR and he also do fun things like adding stickers like the Stormtrooper here. moving along now the burning of the concept of

exclusive design many people ask what is inclusive design will to me. It means being mindful in your choices so that you don't exclude someone accidentally I lack of thought. Do I go to vote here to accessibilities is when I was on Google play music by test engineer came up and I heard device and a speaker like the man. I heard button button button button fancy people smile here cuz you know what this means but I didn't at the time with this means that the application was not accessible. So I asked what is this and she said well, it says your application, but how does someone use

it? This is what this is why I'm here and he was at that moment without realizing it take from zero effort to make you experience for application not inclusive. It only takes a little bit to make an inclusive. So that's what we're here to talk about and accessibility is one of the key lenses. What are most tangible lenses for inclusive design? Why is accessibility important is the right thing to do and to the potentially big business the more than 1 billion people in the world who have some for disability about $0.15 but one in five people in the US will have a disability at some

point in their lives. Just give me another chance at this there more than 300 million people who are color-blind. That's more people are color-blind. There are people in the United States. I think it's important or members expenses accessibility is a spectrum. It's not a binary on-off thing many common ailments. You may not recognize have some potential disability is like you just had a missed call or even a bad back and there still lots of nuances within this grouping of of disabilities. It's more like cups instead of buckets because of their individual people if you have different

abilities people have different kinds of disabilities in the very subtle, but fortunately the things that that help these different groups bring up all the boats up together at the same time. You'll still want to think about situational for temporary impairments like a broken leg or carrying a bag of a bag of groceries or even using your device to invite some all of you think our situation on pyramids and finally different people learn a different speeds either by disability or simply stressed. The course I don't want you to come away with is don't

design for yourself dying for the broadest possible markets and you surprised who you're supporting. How do you do this to ask I'm glad you asked what are ways I talked about. This is a thing called the curb cut. What a curb cut is is that part of the sidewalk that is dipped down. Usually the corner of blocks. This is designed to allow people to wheelchairs to be able to get up onto the sidewalk and that's really profound importance. But it's also really good summer people suitcases for bicycles for baby strollers. So the idea of a curb cut it some of that can help many people

but if propounding impactful profound impact will for people with disabilities another example, if your address please of someone who is hard of hearing you also helping people who forgot the earbuds, Adjusting a v captions. For example, you're helping people forgot the earbuds. You're helping people on a noisy train or conversely in a quiet room when you don't want to make a lot of noise or even non-native speakers this one accommodation count. Lots of people more people than you might realize at the beginning. Another example enters when they're very close to my heart or maybe many of

you here. If you're stuck in an airplane seats, and some of you may have been an airplane quite recently you put on the headsets and something happens behind but you're strapped in you can't get around. What what do you do? Well one cryptitis something that the Google Earth team solved they have to rotate the world around you so you actually can have what's happening behind you be moved into the front. Also YouTube The Artist have solved the same kind of Problem by rotating the world around you. So this way you would improve the world for people in wheelchairs airplane seats or even your comfy

beds. There are few standard today in terms of accessibility around a R&B are there none? So here's an opportunity to really do something if I think they could really impact people's lives. Even before they are people were using devices to help improve the lives of people using their cameras on their cell phones to magnify something with this is actually a device called a magnifier but people didn't realize they were using a sister tag. There was just using your phone to magnify something and it makes

life better. The whole idea. But now with modern AR & VR, you can fly anywhere in the world to someone who is stuck in bed can visit the International Space Station or you can measure the couch in your room. For if the game is his uncle said she was going to ski when someone is shooting from behind you or how about when you're stressed you could have the most important piece of information highlighted or had everything else be ignored for this is what are useful for everybody and fortunately each of these things are also profoundly impactful for people

different kinds of disabilities. Demi Lovato as the AR & VR can help people perhaps the ways you don't expect. Throughout my career. I've had the opportunity to care for children young adults with cancer and some of the difficulties that they have in the anxieties if they develop when I watch them and put on the headsets you can see immediately the change in their body language in their facial expressions. He went and bought art supplies. He took classes that took my mind away from just passively experiencing something. Research is being done today

that shows that VR can actually help people feel less pain and you can change people's lives introduce a new skill or talent. I never knew that I had a wise man once said with great power comes great responsibility. So how do you start? Well one place to start is by with trusted experts for over a decade ablegamers have helped countless Gamers with disabilities either return the games after traumatic injury or start playing games despite of disability. If you could work with Sony and Microsoft that accessibility setting into the platform and they've come up with this

concept called include application, which it comes in a free 48 page PDF and many of the ideas from this talk come from this included vacation Concepts. Assignment a little bit about what works today. So on the right there, you see the new Android o accessibility options and what it looks like many of the settings work right out of the box. Don't worry about trying to read it those kind of small. So here's the highlights. There is a color and contrast adjustment color Corrections for different types of color deficiencies. So my

little brother has trouble differentiating between red and green. He has a hard time telling who the enemy team was the Allied tune, please just shooting around on the phone and the AR & VR apps. We also have mono audio which mixes the right and the left channels into a single Channel and then put those two channels into both earbuds so you can watch videos with just a single everybody and this also works in a RZR. Railside magnification where you triple tap on the screen to zoom in

and this works really well in AR and then we have switch access and TalkBack, which I'll get into. So designing for accessibility things that you can do today. Well switch access and TalkBack requires that have to support Android accessibility API. Are you coming the apis are Java so you can integrate them into your Java or C plus plus apps the switch access makes it possible for people with dexterity impairments to use your app with one or more switch. These switches can take the form of buttons like the volume buttons on the side of the phone or

special sensors like Stephen Hawking had sensors that detected his cheek muscle twitches. So with one button, you can press to cycle through the different items like you see that blue box highlighting and when you get to the one you want you press the second button to select it. And then we have talked back, which is another way you can use to interact with your phone. But I'll let my friend Victor show you how he uses it in a simple Tic Tac Toe app. My name is Rick restaurant and I'm a technical program manager on the Android accessibility Services team

for blind or visually impaired. The way you stole back is by a special touch gestures that I'll put the feedback is provided by speech navigate to screen with Talkback. The first one is called linear navigation and base performed by swiping left and right with a single finger when the element of interest is located double tap on it to activate vibration in this matter. Finger across the elements on the screen. Once the elements of interest is located you double tap on it to activate.

Awesome. I'm glad he won that uses Graphics that are rendered in opengl in C plus plus and allows Android accessibility API to interact with them if you want you can check out the code at github.com Resort. And accessibility should not be an afterthought. This is taken from a recent trip to Japan and in Japan, they have a lot of older houses and businesses. Were they retrofitted with ramp to make it more accessible and this man. I don't know if you can tell or not. It's like a 45 degree angle. So if I even tried to go for it, I would just tip Ford

interesting enough my girlfriend that took this picture as you was trying to get back up that ramp. She just kept slipping I could even get up so not sure what's happening dinner. This is another one. I still don't understand what those are for. So start thinking about it today start testing talking and listening your users have a lot to say test your apps and sub off the mole condition with limited capabilities like with bright sunlight or a noisy environment or holding a cup of coffee in one hand. Understand your

users needs by reaching out to the community, you know reach out to friends or friends of friends people with different accessibility need and listen to what they have to say with an open mind their Valley be a valuable perspective to share. How should you test your apps? And we think that inclusive design should shouldn't just be the job of a single accessibility engineer or a team of Engineers. It should be considered all phases of development from idea to design to develop all the way to lunch. So we've created this pneumonic to make it easy to promote inclusive

design process to all of your developers and the pneumonic is this Saturday morning test. So imagine for a second you woke up on a Saturday morning. If you're groggy you forgot your glasses and your headphones and you just want to sit down and a comfy chair and play some TR. Well this situation simulates a significant portion of different accessibility needs and if you have perfect vision, oral, someone else's glasses try those on with these limitations make sure of these four things that controls are simple. The text is readable. The items are are reachable

and the audio is optional. So again accessibility is an important lens of inclusive design and let's dive into the specifics of how you can apply inclusive design to your immersive applications. There's many opportunities fractions and we can break these down to four big categories of accessibility needs mobility and dexterity visual audio and cognitive. And we will first cover mobility and dexterity needs. This is one of the largest areas of impact and

one of the easiest to make a combination score against these include temporary and permanent disability. So the surface area here is really large. I'll run through a really quickly. So hold on we have people with one hand available or you're just holding a cup of coffee people with shaky hands or you drink too much coffee. You have people with limited finger dexterity with challenges grasping things and pressing buttons or people just wearing gloves. You have people with a hard time bending down grabbing things off the floor work up high off shelves and people of different height.

And then you have people laying down sitting down difficulties turn around or perhaps there in an airplane seat. And this is one of my personal difficulties ever since I started using a wheelchair. Honestly VR has been a lot harder tripping over the cord used to be an issue. It's a lot harder to step over the cord with a wheelchair holding things that moving around. So if I'm holding the two controllers and try to push my chair not really happening and I also have finger dexterity problems were some of the muscles in my hands just don't work. So I have trouble pressing some

buttons like I jump back into this game. I used to play in the enemies are chasing after me and I'm just I can't press that sprint button so casually walking away. Then they didn't end. Well, they're behind me when things are all the way back there. Soap actually this is a good opportunity for us to do a real exercise together. So if I can get your participation, please let's just indulge me for a second and suspend your disbelief Let's Pretend We're Luke Skywalker Jedi in training in a holding a lightsaber in front of you and there's a drone

flying around you shooting lasers that you now you must defend yourself from this drums. Anyways, is that you were before you do it. Let's try to notice the things to happen as you do this experiment. All right. So the drones in front of the lights off in front of the right vodka shots. All right, don't don't feel about your neighbors Jones behind you behind you and then Ella Jones to your left. I'm going to turn to 70° to my left or right and then now the drums back in front of you.

So how did you do? Although it's possible to turn around and block all of his shots and it's difficult. And if you're under a time constraint and makes it impossible to use this application and you want to really test these situations yourself. Don't feel silly. These are very valuable inside. So not only does this apply to me. It's wise to use sitting in these conference Tuesday don't want to whack your neighbors with a lightsaber or the person behind you and you want to look silly sometimes in these public situation and make

sure that these interactions are useful in your app as you do these experiments. Don't Starve until some details on how to make Europe more accessible. So are you saw that example of Google Earth VR allow you to rotate the space around you with the controller noticed how there's also the tunnel vision effect where it blurs out the surroundings and krates this grid that matches your real-world coordinate so you don't get motion sickness and the Google Earth VR also allows you to move places with the controller. This is super valuable for someone trying to move around without actually

having the space or sitting in a chair and it's also valuable to give users to control of having incremental vs. Smooth rotations and movements. This means snapping those rotations 215 degrees increments versus smoothly going around or jumping forward vs. Smoothly gliding Ford and this provides users of choice of comfort vs. Finer grain control. And then let's talk about Heights. There's users of different heights and it's super common in VR, especially with little kids these days getting really excited about this new

technology. My little sister loves job simulator when it first came out you played it all the time, but she couldn't reach some of the items off the high shelves. She's a 13 year old girl without the stall. So one day we discover that the end of this new shorter human mode at the launch screen you turn it on and the world shrinks down into a doll house. So you're feeling like a 10-foot giant inside this house. Then you can just reach everything with ease. Then she's like 50 something generally you should designed to minimize for reaching down low or up high

perhaps using a magnetic hand to grab things would be one way to solve this and interesting me enough with three degrees of freedom controllers, like on the Daydream View and the Mirage solo the hardware allows for more accessible experiences. There's no need to reach up or down. And custom controls allowing users to choose what works best for them. Soap for me I having the ability to remap that sprint button. I can finally use that game again X-Play and I can run away from enemies and giving users assistant controls. There's another way to help some UI buttons

are just too small or not obvious that they're interactable. So using the spendy laser is one way to solve that problem and shaky hands every morning coffee won't be a problem anymore. Either a lot of these concepts are borrowed from gaming like assisted aiming or assisted driving. These are well-established interactions in console games already ending review super valuable for VR experiences as well. And finally sensitivity adjustments on different controls the joystick rotations Etc. It allows users to choose between finer-grained control versus faster movement.

And I was talk about mobile a our interaction has a lot from mobile itself. The interactions are well-defined for 2 DUI, but for 3 DUI like this you want to have a full size and rotation four different objects. This is another great curb good opportunity and they are as it allows for people to use their app and small spaces, but it profoundly impact those using your app stuck in a chair. And for me, I can't really see up high or or reach very far and so I can move the object displaces that works best for me and having large tap Target food sources for simpler user

interfaces friendlier UI and also makes text more readable. Generally you want the top targets to be at least like finger size. And finally having alternative gestures so I don't turn Dave to swiping or shaking the phone having a extra tap Target to do these interactions with easeful. Don't have visual impairments. It was you look towards visual impairments. Look at the groups of people that you'll be supporting our people without glasses people with glasses people high contrast these or public efficiencies Knox. Stopping people with visual Peridot blinds discovers

an awful lot of people to a different phases of your life. What are the most important things you can do is have angle hear me say that's a couple of times multimodal affordances. This is about having audio and text at the same time. If you have a narrative haven't voiced over and written down this is helping people with both audio pyramids and visual impairments the next to support a large text a large fonts. That is so much easier to see if you can and the next one so does this is a contrary good contrast and trim Wy you probably can't read all of the letters there

because I intentionally made the contrast badge reproduce the situation to the conference is bad. It don't matter people can't see it. If I finally if you want to support me with color blindness don't use only color to support things like health or progress. Please use dedicated icon or Textures in addition to the color because this way this a different way they can tell the difference and always please try to use a color-blind safe pallets. A lot of people ask what I talk about. I do work at accessibility and a r b r u colossal how do blind people use AR or VR? And if they will write down

it's really hard and hopefully this is something that we all can work on together cuz at this time there's no support in the major game and gems for accessibility Services. What happens is the UI in three-dimensional space are literally just text you and hanging out there with a script attached. There's no such thing as a a VOR check box or text box is just a floating texture as the script allows it to act like a text box. There's no window manager is no way for the accessibility services to understand. How did it navigates what new things are so this is work that needs to get

done. Episode.com be true for VR or 3D with an AR. This is actually true within all 3D games as well. So to that end, please don't forget the accessibility Services. Gluco. He mentioned a little while ago from the store get hub.com Resort. Yes, and if you do want this, please reach out to the game engine manufacturers, then gave me to developers. The laws of indie game developers is not just begun Holly Duo reach out ask for it so they can do it. Louboutin towards hearing impairments Here you were helping people with impairments

hearing impairments. But also again, it's the noisy environment with a quiet environments for people without their headphones here. Again, this is that multimodal affordance the audio and video happening at the same time captions and duration it the same idea but slightly different instances it captions. It's normally video button duration. It's Pepsi to the game or just any kind of conversation. Please include the captions or narration. And do you think so everyone can enjoy the store that you're building and often if you have avatars in other VR or a are we find

captions are often connected like a word bubble or a thought bubble? That's pretty useful as long as people know where to look. So be careful with that. but in terms of captions This is really important to have unfortunately, not all platforms support closed captioning in VR YouTube VR has always include Heather with the for the captions and the way YouTube. ER. They just called so it's a little black box with white letters as you turn your head it spins around with you because there's no way for us to tell where for it to go. At least it's there. But

again, not all VR platform support this so if you have video or if you're using Video Edit application, please turn it into the video. It's not the best of the best of the justice for captions. But if you can't please at least do this and when you're doing it with her to standards that are most common one is a three up like you see here. It's twelve 4 and 7 and the idea here is to make sure there's always some bit of the caption was available in the peripheral vision. So, you know that this nothing happening what's happening at becoming more common is 4 up soda north

south east west or all the things types of a cube put your put your captions in these places and everyone should be able to find them. What is this new format called video in VR 180 and we just announced with a low Mirage. This is a great camera and the way to make sure that you have your captchas when you go into YouTube put everything in a 16 by 9 crop and everything should be visible there and that works just fine. movie Gone towards cognitive impairments the people that you help here are distracted users like parents. People with this account difficult to

remember to remember your understanding language like parents or for language speakers. And what you do here is you want to make your you wines simple and easy to use that this isn't your Thoreau simplify simplify simplify. This is think about the use of your video the uses of your idea. Why how do you guide someone from one idea to the next make it in to it? If not, just beautiful. If you're talking about the situation, we have levels the three things here. That would be better useful. One is have a Sandbox level for people who have a hard time learning or or hard time manipulating

controls give them a place where they can mess around and learn how to use your game or your experience without having an impact their Village actually play the game with a four words. If you can't do that, make sure you have a good training to give people a safe place an opportunity to learn how to build a how to play your game how to use your interaction and then finally have different difficulty modes the standard easy medium hard, but you don't want to be mean you don't tell anyone you're going easy on them. So I really like this story to get balanced and

challenge that some game was up front about it's a lot nicer but I get the same idea if you want someone to hear your story. Don't stop them let them play to their ability. And finally, sometimes people just can't get through the level. So why should they be stopped in the process of getting through your story in this type of case come up with an auto Palace say if you can't get past it in 10 times you want to move on to the next level so they can see what your story is sometimes to say. Yes sometimes to say no, but why prevent them from having the choice to Auto passes are really nice way to

help people understand and finish the work that you've done. Now taking the Step Up from swim at the game environment, but an hour ago, we just announced my team in the sentence. His ability has just announced an application called look out what this is. It's an AR application that uses the camera and and the uses of visual detection of the world around you and it provides him an audio description of the world around you so it in the end is here we have on the on the left. We have a picture of lavender as a caption that says lavender and what

you would hear is lavender at 12. On the right hand side. We have a series of contests. so the important thing is a 3 things I want to talk about here that are important that context menu is actually surprisingly important because unless we know your context we don't know what to say or more importantly. We don't know what not to say. If we don't know what you care about will read everything to you. If I had this right here idly person person person person person person person person and it's all correct, but it doesn't matter. So providing the context gives you an

opportunity that just say the things that matter in real-time. The second thing is blind user use of the camera very differently than you expect. It's really true. What are testing all of our Engineers because they sided with General in localized the camera on the thing that they wanted to have recognized get we went into user testing. But why did we see what they different kinds of things? So what we ended up having to do that actually capture all of the video if things are moving around Stitch it together and go very very large image and then going to see are all of it. Then we

finally got the results that we were looking for him finally. Faithfulness, where are things in space matters a lot. If I'm describing to that there was a lavender in front of you and makes a big difference to visit 10 to 4:12. So when will you try the guide summon to something that sense of space and where things are is really important in terms of understanding the world around you. Same conclusion. Now, please. Don't forget these key points one don't design for yourself or your situation to

test talk and listen sit down and take the Saturday morning test check that controls are simple, Texas readable items are reachable audio is optional and think about the different types of knee from Mobility visual auditory and cognitive. What are your curb Cuts? What is Tempe that you can do that helps everybody but has a profound impact on someone with disabilities. And you remember nothing else? You can always go to ablegamers and complication. This is a really great resource if you want like a one-stop-shop for everything. We want to hear from you provide

feedback on the session as I link. and thank you so much for attending our talk. Thank you.

Cackle comments for the website

Buy this talk

Access to the talk “Accessibility for AR and VR”
Available
In cart
Free
Free
Free
Free
Free
Free

Access to all the recordings of the event

Get access to all videos “2018 Google I/O”
Available
In cart
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “Software development”?

You might be interested in videos from this event

September 28, 2018
Moscow
16
159
app store, apps, development, google play, mobile, soft

Similar talks

Tom Salter
Tech Lead at Google
Available
In cart
Free
Free
Free
Free
Free
Free
Dave Burke
Software Engineer at Google
+ 6 speakers
Romain Guy
Senior Staff Software Engineer at Google
+ 6 speakers
Chet Haase
Leads the Android Toolkit team at Google
+ 6 speakers
Dianne Hackborn
Product Manager at Google
+ 6 speakers
Aurash Mahbod
Director of Engineering at Google
+ 6 speakers
Tor Norbye
Tech lead for Android Studio at Google
+ 6 speakers
Stephanie Saad
Software Engineer at Google
+ 6 speakers
Available
In cart
Free
Free
Free
Free
Free
Free
Stephan Herhut
Software Engineer at Google
Available
In cart
Free
Free
Free
Free
Free
Free

Buy this video

Video

Access to the talk “Accessibility for AR and VR”
Available
In cart
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
558 conferences
22059 speakers
8245 hours of content